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Introduction 





Docker, made available as open source in March 2013, has become the de facto containerization platform. 
The Docker Engine by itself does not provide functionality to create a distributed Docker container cluster 

or the ability to scale a cluster of containers, schedule containers on specific nodes, or mount a volume. The 
book is about orchestrating Docker containers with the Docker-native Swarm mode, which was introduced 
July 2016 with Docker 1.12. Docker Swarm mode should not be confused with the legacy standalone Docker 
Swarm, which is not discussed in the book. The book discusses all aspects of orchestrating/managing Docker, 
including creating a Swarm, using mounts, scheduling, scaling, resource management, rolling updates, load 
balancing, high availability, logging and monitoring, using multiple zones, and networking. The book also 
discusses the managed services for Docker Swarm: Docker for AWS and Docker Cloud Swarm mode. 


Docker Swarm Design Patterns 


“A software design pattern is a general reusable solution to a commonly occurring problem within a given 
context in software design.” (Wikipedia) 

Docker Swarm mode provides several features that are general-purpose solutions to issues inherent in 
a single Docker Engine. Each chapter starting with Chapter 2 introduces a problem and discusses a design 
pattern as a solution to the problem. 


Why Docker Swarm Mode? 


Why use the Docker Swarm mode when several container cluster managers are available? Docker Swarm 
mode is Docker-native and does not require the complex installation that some of the other orchestration 
frameworks do. A managed service Docker for AWS is available for Docker Swarm to provision a Swarm 
on production-ready AWS EC2 nodes. Docker Cloud may be linked to Docker for AWS to provision a 

new Swarm or connect to an existing Swarm. Docker 1.13 includes support for deploying a Docker Stack 
(collection of services) on Docker Swarm with Docker Compose. 


What the Book Covers 


Chapter 1 introduces running a Docker standalone container on CoreOS Linux. The chapter establishes the 
basis of the book and subsequent chapters discuss how the management design patterns provided by the 
Swarm mode solve problems inherent in a standalone Docker Engine. 

Chapter 2 introduces the Swarm mode, including initializing a Swarm and joining worker nodes to 
the Swarm. Chapter 2 includes promoting/demoting a node, making a node (manager or worker) leave a 
Swarm, reinitializing a Swarm, and modifying node availability. 
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Chapter 3 discusses the managed service Docker for AWS, which provisions a Docker Swarm by 
supplying the Swarm parameters, including the number of managers and workers and the type of EC2 
instances to use. AWS uses an AWS CloudFormation to create the resources for a Swarm. Docker for AWS 
makes it feasible to create a Swarm across multiple AWS zones. 

Chapter 4 is about Docker services. Two types of services are defined—replicated and global. Chapter 4 
discusses creating a service (replicated and global), scaling a replicated service, listing service tasks, and 
updating a service. 

Chapter 5 discusses scaling replicated services in more detail, including scaling multiple services 
simultaneously. Global services are not scalable. 

In Chapter 6, two types of mounts are defined: a bind mount and volume mount. This chapter discusses 
creating and using each type of mount. 

Chapter 7 is about configuring and using resources in a Swarm. Two types of resources are supported 
for configuration: memory and CPU. Two types of resource configurations are defined: reserves and limits. 

It discusses creating a service with and without resources specification. 

Chapter 8 discusses scheduling service tasks with the default and custom scheduling. Scheduling 
constraints are also discussed. 

Chapter 9 discusses rolling updates, including setting a rolling update policy. Different types of rolling 
updates are provisioned, including updating to a different Docker image tag, adding/removing environment 
variables, updating resource limits/reserves, and updating to a different Docker image. 

Chapter 10 is about networking in Swarm mode, including the built-in overlay networking called ingress 
and support for creating a custom overlay network. 

Chapter 11 is about logging and monitoring in a Swarm, which does not provide a built-in support for 
logging and monitoring. Logging and monitoring is provided in a Swarm with a Sematext Docker agent, 
which sends metrics to a SPM dashboard and logs to a Logsene user interface and Kibana. 

Chapter 12 discusses load balancing across service tasks with ingress load balancing. An external AWS 
elastic load balancer may also be added for distributing client requests across the EC2 instances on which a 
Swarm is based. 

Chapter 13 discusses developing a highly available website that uses an Amazon Route 53 to create a 
hosted zone with resource record sets configured in a Primary/Secondary failover mode. 

Chapter 14 discusses another managed service, Docker Cloud, which may be used to provision a 
Docker Swarm or connect to an existing Swarm. 

Chapter 15 discusses Docker service stacks. A stack is a collection of services that have dependencies 
among them and are defined in a single configuration file for deployment. 


Who this Book Is For 


The primary audience of this book includes Docker admins, Docker application developers, and Container 
as a Service (CaaS) admins and developers. Some knowledge of Linux and introductory knowledge of 
Docker—such as using a Docker image to run a Docker container, connecting to a container using a bash 
shell, and stopping and removing a Docker container—is required. 
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CHAPTER 1 


Getting Started with Docker 





Docker has become the de facto containerization platform. The main appeal of Docker over virtual 
machines is that it is lightweight. Whereas a virtual machine packages a complete OS in addition to the 
application binaries, a Docker container is a lightweight abstraction at the application layer, packaging 
only the code and dependencies required to run an application. Multiple Docker containers run as isolated 
processes on the same underlying OS kernel. Docker is supported on most commonly used OSes, including 
several Linux distributions, Windows, and MacOS. Installing Docker on any of these platforms involves 
running several commands and also setting a few parameters. CoreOS Linux has Docker installed out- 
of-the-box. We will get started with using Docker Engine on CoreOS in this chapter. This chapter sets the 
context of the subsequent chapters, which discuss design patterns for managing Docker Engine using the 
Swarm mode. This chapter does not use Swarm mode and provides a contrast to using the Swarm mode. 
This chapter includes the following sections: 


e Setting the environment 


e Running a Docker application 


Setting the Environment 


We will be using CoreOS on Amazon Web Services (AWS) EC2, which you can access at https : //console. 
aws.amazon.com/ec2/v2/home?region-us-east-1it. Click on Launch Instance to lauch an EC2 instance. 
Next, choose an Amazon Machine Image (AMI) for CoreOS. Click on AWS Marketplace to find a CoreOS 
AMI. Type CoreOS in the search field to find a CoreOS AMI. Select the Container Linux by CoreOS (Stable), 
as shown in the EC2 wizard in Figure 1-1, to launch an instance. 


1. Choose AMI 2. Choose Instance Type Configure instance 


Cancel and Exit 


step 1: Choose an Amazon Machine Image (AMI) 


An AMI is a template that contains the software configuration (operating system, application server, and applications) required to launch your instance. You can select an AMI 


provided by AWS, our user community, or ihe AWS Marketplace: or you can select one of your own AMIS 
Quick Start 1 to 6 of 6 Products 
Ci coreos 
My AMIS 


container linus Container Linux by CoreOS (Stable) —— 
se st teeter (0)1 1225.9.0 Previous versions | Sold by Core0S 


Community AMIS 50.00/hr for software + ANS usage fees 
| Free tier egiie | Linun/Uinix, Other 1235.9.0 | 64-6 Amazon Machine Image (AMI) | Updated: 2/8/17 


* Categories CoreO'S Container Linux automates software updates to ensure better security and reliability of machines 





AWS Marketplace 


and containers running on large-scale clusters. Operating system updates 
All Categories 
More into 
Software Infrastructure (6) 


Figure 1-1. Selecting an AMI for CoreOS Linux 


O Deepak Vohra 2017 ] 
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From Choose an Instance Type, choose the t2.micro Type and click on Next. In Configure Instance 
Details, specify the number of instances as 1. Select a network or click on Create New VPC to create a new 
VPC. Select a subnet or click on Create New Subnet to create a new subnet. Select Enable for Auto-Assign 
Public IP. Click on Next. 

From Add Storage, select the default settings and click on Next. In Add Tags, no tags need to be added. 
Click on Next. From Configure Security Group, add a security group to allow all traffic of any protocol in all 
port ranges from any source (0.0.0.0/0). Click on Review and Launch and subsequently click on Launch. 

Select a key pair and click on Launch Instances in the Select an Existing Key Pair or Create a New Key 
Pair dialog, as shown in Figure 1-2. 
















select an existing key pair or create a new key pair x 


A key pair consists of a public key that AWS stores, and a private key file that you store. Together. 
they allow you to connect to your instance securely. For Windows AMIs, the private key file is required 
to obtain the password used to log into your instance. For Linux AMIs, the private Key file allows you to 
securely SSH into your instance. 


Note: The selected key pair will be added to the set of keys authorized for this instance. Learn more 
about removing existing key pairs from a public AMI. 


Choose an existing key pair Y 
Select a key pair 
coreos T 


9 | acknowledge that | have access to the selected private key file (coreos.pem), and that 
without this file, | won't be able to log into my instance. 






Cancel Launch Instances 


Figure 1-2. Launch instances 
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An EC2 instance with CoreOS is launched. Obtain the public DNS or IPv4 public IP address of the EC2 
instance from the EC2 Console, as shown in Figure 1-3, to SSH login into the instance. 


Launch Instance Connect Actions ¥ 
EB Hench instance | o 8 © 


Q, search : j-09dd406018d232147 vdd filt o 1 to 1 of 1 
ig Name * Instance ID ^ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 
@ Coros i-09dd4e6018d2321 t2.micro us-east-le @ running © 2/2 checks None `a ec 2-54-196-51-20.com.. 
Instance: [ i-09dd4e6018d232147 (CoreOS) Public DNS: ec2-54-196-51-20.compute-1.amazonaws.com mpm mm 
Description Status Checks Monitoring Tags Usage Instructions 
Instance ID -094d4e60184d232147 Public DNS (IPv4)  ec2-54-196-51-20 compute 
lLamazonaws.com 
Instance state — running Nw IPvá Pubic IP — 54.196. 51.20 
s 2J r 
Instance type t2 micro IPv6 IPs 


Figure 1-3. Public DNS and public IPv4 


SSH login into the EC2 instance as user "core" 


ssh -i "coreos.pem" core@<public ip» 


Running a Docker Application 


As mentioned earlier, Docker is pre-installed on CoreOS. Run the docker command to list its usage, as 
shown in the following bash shell: 


core@ip-172-30-4-75 ^ $ docker 
Usage: docker [OPTIONS] COMMAND [arg...] 


docker [ --help | -v | --version | 
A self-sufficient runtime for containers. 
Options: 
--config-^/.docker Location of client config files 
-D, --debug Enable debug mode 
-H, --host=[ ] Daemon socket(s) to connect to 
-h, --help Print usage 
-l, --log-level=info Set the logging level 
--tls Use TLS; implied by --tlsverify 


--tlscacert-^/.docker/ca.pem Trust certs signed only by this CA 
--tlscert-^/.docker/cert.pem Path to TLS certificate file 


--tlskey-^/.docker/key.pem Path to TLS key file 
--tlsverify Use TLS and verify the remote 
-V, --Version Print version information and quit 
Commands : 
attach Attach to a running container 
build Build an image from a Dockerfile 
commit Create a new image from a container's changes 
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cp Copy files/folders between a container and the local filesystem 
create Create a new container 
diff Inspect changes on a container's filesystem 


Output the Docker version using the docker version command. For native Docker Swarm support, the 
Docker version must be 1.12 or later as listed in the bash shell output. 


coreQip-172-30-4-75 ~ $ docker version 
Client: 

Version: 1.12.6 

API version: 1.24 

Go version: g01.7.5 

Git commit:  a82d35e 


Built: Mon Jun 19 23:04:34 2017 
OS/Arch: ]inux/amd64 

Server: 

Version: 1.12.6 


API version: 1.24 

Go version: g01.7.5 

Git commit:  a82d35e 

Built: Mon Jun 19 23:04:34 2017 
OS/Arch: ]inux/amd64 


Run a Hello World app with the tutum/hello-world Docker image. 
docker run -d -p 8080:80 --name helloapp tutum/hello-world 
The Docker image is pulled and a Docker container is created, as shown in the following listing. 


core@ip-172-30-4-75 ~ $ docker run -d -p 8080:80 --name helloapp tutum/hello-world 
Unable to find image 'tutum/hello-world:latest' locally 

latest: Pulling from tutum/hello-world 

658bc4dc7069: Pull complete 

a3ed95caebo2: Pull complete 

af3cc4b92fa1: Pull complete 

d0034177ece9: Pull complete 

983d35417974: Pull complete 

Digest: sha256:0d57def8055178aafb4c7669cbc25ec17 f0acdab97cc587F30150802da8F8d85 
Status: Downloaded newer image for tutum/hello-world:latest 
1b7a85df6006b41ea1260b5ab957113c9505521cc8732010d663a5e236097502 


List the Docker container using the docker ps command. 


coreQip-172-30-4-75 ~ $ docker ps 

CONTAINER ID IMAGE COMMAND CREATED STATUS 

PORTS NAMES 

1b7a85df6006 tutum/hello-world ^ "/bin/sh -c 'php-fpm " 19 minutes ago Up 19 minutes 
0.0.0.0:8080->80/tcp X helloapp 
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The port mapping for the Docker container is also listed using the docker ps command, but it may also 
be obtained using the docker port <container> command. 


core@ip-172-30-4-75 ~ $ docker port helloapp 
80/tcp -> 0.0.0.0:8080 


Using the 8080 port and localhost, invoke the Hello World application with curl. 
curl localhost:8080 
The HTML markup for the Hello World application is output, as listed shown here. 


core@ip-172-30-4-75 ~ $ curl localhost:8080 


<html> 
<head> 
<title>Hello world!</title> 
«link href='http://fonts.googleapis.com/css?family=Open+Sans: 400, 700° 
rel-'stylesheet' type='text/css'> 
<style> 
body { 
background-color: white; 
text-align: center; 
padding: 50px; 
font-family: "Open Sans","Helvetica Neue" ,Helvetica, Arial, sans-serif; 
j 
#logo { 
margin-bottom: 40px; 
j 
</style> 
</head> 
<body> 
<img id-"logo" src="logo.png" /» 
<h1>Hello world! </h1> 
«h3»My hostname is 1b7a85df6006«/h3» 
</body> 
</html> 


Using the public DNS for the EC2 instance, the Hello World application may also be invoked in a 
browser. This is shown in the web browser in Figure 1-4. 
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tutum 


Hello world! 


My hostname is 1b7a85df6006 





Figure 1-4. Invoking the Hello World application in a web browser 


The docker stop <container> command stops a Docker container. The docker rm <container> 
command removes a Docker container. You can list Docker images using the docker images command. 
A Docker image may be removed using the docker rmi <image> command. 


core@ip-172-30-4-75 ~ $ docker stop helloapp 


helloapp 

core@ip-172-30-4-75 ~ $ docker rm helloapp 

helloapp 

core@ip-172-30-4-75 ~ $ docker images 

REPOSITORY TAG IMAGE ID CREATED SIZE 
tutum/hello-world latest 31e17b0746e4 19 months ago 17.79 MB 


core@ip-172-30-4-75 ~ $ docker rmi tutum/hello-world 

Untagged: tutum/hello-world: latest 

Untagged: tutum/hello-world@sha256:0d57def8055178aafb4c7669cbc25ec17f0acdab97cc587F30150802da8f8d85 
Deleted: sha256:31e17b0746e48958b271d3dd4fe179fbba7e8efe14ad7a51e964181a92847a6 
Deleted: sha256:e1bc9d364d30cd2530cb673004dbcdf1eae0286e41a0fb217dd14397bf9debc8 
Deleted: sha256:a1f3077d3071bd3eed5bbe5c9c036F15ce3f6b4b36bdd7 7601f8b8F03c6f874F 
Deleted: sha256:ff7802c271f507dd79ad5661ef0e8c7321947c145f1e3cd434621fa869fa648d 
Deleted: sha256:e38b71a2478cad712590a0eace1e08f100a293ee19a181d5f5d5a3cdb0663646 
Deleted: sha256:5f27c27cccó6daedbc6ee05562$f96f719d7f0bb38d8e95b1c1f23bb9696439916 
Deleted: sha256:fab20b60d8503ffO0bc94ac3d25910d4a10f366d6da1f69ea53a05bdef469426b 
Deleted: sha256:a358990fe25749e088fd9a9d2999c9a17b51921eb3f7df925a300205207a3172b08 
coreQip-172-30-4-75 ~ $ 
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Summary 


This chapter sets the basis for subsequent chapters by using a single Docker Engine on CoreOS. Subsequent 
chapters explore the different design patterns for managing distributed Docker applications in a cluster. The 
next chapter introduces the Docker Swarm mode. 







CHAPTER 2 


Using Docker in Swarm Mode 





The Docker Engine is a containerization platform for running Docker containers. Multiple Docker 
containers run in isolation on the same underlying operating system kernel, with each container having its 
own network and filesystem. Each Docker container is an encapsulation of the software and dependencies 
required for an application and does not incur the overhead of packaging a complete OS, which could 

be several GB. Docker applications are run from Docker images in Docker containers, with each Docker 
image being specific to a particular application or software. A Docker image is built from a Dockerfile, with 
a Dockerfile defining the instruction set to be used to download and install software, set environment 
variables, and run commands. 


The Problem 


While the Docker Engine pre-1.12 (without native Swarm mode) is well designed for running applications in 
lightweight containers, it lacks some features, the following being the main ones. 


e No distributed computing—No distributed computing is provided, as a Docker 
Engine is installed and runs on a single node or OS instance. 


e No fault tolerance—As shown in the diagram in Figure 2-1, if the single node on 
which a Docker Engine is running fails, the Docker applications running on the 
Docker Engine fail as well. 


X. Docker Engine 





Figure 2-1. Single node Docker cluster 
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The Solution 


With Docker Engine version 1.12 onward, Docker container orchestration is built into the Docker Engine 

in Swarm mode and is native to the Docker Engine. Using the Swarm mode, a swarm (or cluster) of nodes 
distributed across multiple machines (OS instances) may be run in a master/worker/ pattern. Docker Swarm 
mode is not enabled in the Docker Engine by default and has to be initialized using a docker command. 
Next, as an introduction to the Docker Swarm mode, we introduce some terminology. 


Docker Swarm Mode 


Docker Swarm is a cluster of Docker hosts connected by an overlay networking for service discovery. 

A Docker Swarm includes one or more manager nodes and one or more worker nodes, as shown in 

Figure 2-2. In the Swarm mode, a Docker service is the unit of Docker containerization. Docker containers 
for a service created from a Manager node are deployed or scheduled across the cluster and the Swarm 
includes a built-in load balancing for scaling the services. The expected state for a service is declared on 
the manager, which then schedules the task to be run on a node. However, the worker node itself still pulls 
the image and starts the container. 


Docker 
Swarm 
Mode 


Node Node Node 
Docker Docker Docker 
Engine Engine Engine 


Worker Worker Worker 
Node Node Node 
Docker Docker Docker 
Engine Engine Engine 





Figure 2-2. Docker Swarm mode cluster 


Nodes 


An instance of a Docker host (a Docker Engine) is called a node. Two types of node roles are provided: 
manager nodes and worker nodes. 
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Service 


A service is an abstraction for a collection of tasks (also called replicas or replica tasks) distributed across 

a Swarm. As an example, a service could be running three replicas of an Nginx server. Default scheduling, 
which is discussed in Chapter 7, uses the “spread” scheduling strategy, which spreads the tasks across 

the nodes of the cluster based on a computed node rank. A service consists of one or more tasks that run 
independent of each other, implying that stopping a task or starting a new task does not affect running other 
tasks. The Nginx service running on three nodes could consist of three replica tasks. Each task runs a Docker 
container for the service. One node could be running multiple tasks for a service. A task is an abstraction for 
the atomic unit of scheduling, a “slot” for the scheduler to run a Docker container. 


Desired State of a Service 


The “desired state” of a service refers to the service state as defined in the service definition when creating 
the service. As an example, a service definition could define a service as consisting of three replicas of an 
Nginx server. 


Manager Node and Raft Consensus 


When the Swarm is first created, the current node becomes the first manager node. By default, all manager 
nodes are also workers. The manager node performs the cluster orchestration and manages the Swarm, 
including the initial scheduling of service tasks and subsequent reconciliation, if any, between the desired 
state and the actual state of services. As an example, for a service definition consisting of three replicas of an 
Nginx server, the manager node would create three tasks and schedule the tasks on Swarm worker nodes in the 
Swarm. Subsequently, if a node running a task were to fail, the Swarm manager would start a new replacement 
task on the worker nodes still in the Swarm. The Swarm manager accepts the service definition when a service 
is created and schedules the service on one or more worker nodes as service tasks. The Swarm manager node 
also manages the scaling of service by adding/removing service tasks. The Swarm manager assigns each service 
aunique DNS name and starts Docker containers via service replica tasks. The manager node monitors the 
cluster state. The Swarm manager is also a worker node by default, which is discussed in the next section. 

To refer to “the manager node" is actually a simplification of the Swarm Manager, as a Swarm may 
consist of one or more manager nodes. Each manager node keeps the complete cluster state data, including 
which service replica tasks are running on which node and the node roles, and participates in Swarm 
management for the Raft consensus. The Raft consensus is merely an algorithm to create decisions/ 
agreements (consensus) within a group in a distributed fashion. Swarm uses it to make decisions such 
as leader elections, cluster membership, service changes, etc. In the Swarm mode, Raft consensus is 
an agreement among the manager nodes for a global cluster state parameter such as about the state 
of data value stored in a database. Swarm managers share data using Raft. Raft consensus is a protocol 
for implementing distributed consensus among all the reachable manager nodes in a Swarm. The Raft 
Consensus Algorithm has several implementations and its implementation in the Swarm mode has the 
properties typically found in distributed systems, such as the following: 


e Agreement of values for fault tolerance 
e Cluster membership management 
e Leader election using mutual exclusion 


Only one manager node, called the /eader, performs all the cluster orchestration and management. Only 
the leader node performs the service scheduling, scaling, and restarting of service tasks. The other manager 
nodes are for the fault tolerance of Swarm manager, which implies that if the leader node were to fail, one of 
the other manager nodes would be elected as the new leader and take over the cluster management. Leader 
election is performed by a consensus from the majority of the manager nodes. 
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Worker Nodes 


A worker node actually runs the service replica tasks and the associated Docker containers. The 
differentiation between node roles as manager nodes and worker nodes is not handled at service 
deployment time but is handled at runtime, as node roles may be promoted/demoted. Promoting/demoting 
a node is discussed in a later section. Worker nodes do not affect the manager Raft consensus. Worker 

nodes only increase the capacity of the Swarm to run service replica tasks. The worker nodes themselves do 
not contribute to the voting and state held in the raft, but the fact that they are worker nodes is held within 
the raft. As running a service task requires resources (CPU and memory) and a node has a certain fixed 
allocatable resources, the capacity of a Swarm is limited by the number of worker nodes in the Swarm. 


Quorum 


A quorum refers to agreement among the majority of Swarm manager nodes or managers. If a Swarm loses 
quorum it cannot perform any management or orchestration functions. The service tasks already scheduled 
are not affected and continue to run. The new service tasks are not scheduled and other management 
decisions requiring a consensus, such as adding or removing a node, are not performed. All Swarm 
managers are counted toward determining majority consensus for fault tolerance. For leader election only 
the reachable manager nodes are included for Raft consensus. Any Swarm update, such as the addition or 
removal of a node or the election of a new leader, requires a quorum. Raft consensus and quorum are the 
same. For high availability, three to five Swarm managers are recommended in production. An odd number 
of Swarm managers is recommended in general. Fault tolerance refers to the tolerance for failure of Swarm 
manager nodes or the number of Swarm managers that may fail without making a Swarm unavailable. 
Mathematically, “majority” refers to more than half, but for the Swarm mode Raft consensus algorithm, Raft 
tolerates (N-1)/2 failures and a majority for Raft consensus is determined by (N/2)+1. N refers to the Swarm 
size or the number of manager nodes in the Swarm. 


Swarm Size = Majority + Fault Tolerance 


As an example, Swarm sizes of 1 and 2 each have a fault tolerance of 0, as Raft consensus cannot be 
reached for the Swarm size if any of the Swarm managers were to fail. More manager nodes increase fault 
tolerance. For an odd number N, the fault tolerance is the same for a Swarm size N and N41. 

As an example, a Swarm with three managers has a fault tolerance of 1, as shown in Figure 2-3. Fault 
tolerance and Raft consensus do not apply to worker nodes, as Swarm capacity is based only on the worker 
nodes. Even if two of the three worker nodes were to fail, one Worker node, even if the manager nodes are 
manager-only nodes, would keep the Swarm available though a reduction in Swarm capacity and could 
transition some of the running tasks to non-running state. 
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Node Node Node 
Docker 
Engine 


Docker 
Engine 





Figure 2-3. Fault tolerance for a Swarm 


This section covers the following topics: 
e Setting the environment 
e Initializing the Docker Swarm mode 
e Joining nodes to the Swarm cluster 
e Testing the Swarm cluster 
e Promoting a worker node to manager 
e Demoting a manager node to worker 
e Making a worker node leave the Swarm cluster 
e Making A worker node rejoin the Swarm cluster 
e Making a manager node leave the Swarm cluster 
e  Reinitializing a Swarm 
e Modifying node availability 


e Removing a node 
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Setting the Environment 


This chapter shows you how to create a three-node Swarm consisting of one manager node and two worker 
nodes. Create three Amazon EC2 instances using CoreOS Stable AMI, as shown in the EC2 console in 
Figure 2-4. Enable all traffic between the EC2 instances when configuring the security group for the EC2 
instances. Obtain the IP address of the EC2 instance started for the Swarm manager. 


L h Insta c t Acti a : 


CL Instance State : Running Add filter o 1to030f3 
Mame - Instance ID * Instance Type ~ Availability Zone ~ Instance State ~ ‘Status Checks ~ — Alarm Status Public DNS [(IPy4) 

B SwamManager KÜTbi2315cb7c833be = I2. micro us-east-1f @ running © 2/2 checks None Meo 06c2-34-204-168-217 co. 
SwarmWv/orker 1-053363 22817555 t2 micro us-east-1f @ running @ 2/2 checks Mane Y ee 2-34-204-199-45 co 
SwarmWorker i584 3beeebed 30d t2 micro us-Bast- Tf Q running © 22 checks None Me 80?-34-231-70-10.com 

Instance: | i-01b12215cb7c823be (SwarmManager) Public DNS: ec2-34-204-168-217.compute-1.amazonaws.com m mp rj 

Description Status Checks Monitoring Tags Usage Instructions 
instance ID — i-D1b12315cb7c 333be Public DNS (IPvd) — &c2-34-204-168-217 compute- 
i amazonaws.com 
Instance stale running & IPvd Public IP — 34.204. 168 217 
instance type — t2. micro pi IFG IPs 
Elastic IFs Private ONS — ip-172-30-5-70 ec 2 internal 
Availabilty zone — us-east-1f Private IPs 172.30.5 70 
Security groups Container Linux by Cones -Stable—-1409-7- Secondary private IPs 
Ü-AutogenByAWSMP-1. view inbound rules 
Scheduled events Mo scheduled events VRC ID — vpc-dcÜübobab 


Figure 2-4. EC2 instances 


Initializing the Docker Swarm Mode 


Docker Swarm mode is not enabled by default and needs to be enabled. SSH login to the EC2 instance 
started for the Swarm manager using the public IP address. 


ssh -i "coreos.pem" core@34.204.168.217 


Docker Swarm mode is available starting with Docker version 1.12. Verify that the Docker version is at 
least 1.12 using the docker --version command. 


[root@localhost ~]# ssh -i "coreos.pem" core@34.204.168.217 
Container Linux by CoreOS stable (1409.7.0) 
coreQip-172-30-5-70 ~ $ docker --version 

Docker version 1.12.6, build a82d35e 


To initialize the Swarm, use the docker swarm init options command. Some of the options the 
command supports are listed in Table 2-1. 
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Table 2-1. Command Swarm init Options 


Option 


--advertise-addr 


--availability 


--force-new-cluster 


--listen-addr 


Description Default Value 


Advertised address in the format <ip|interface>[: port]. 
The advertised address is the IP address at which other nodes 
may access the Swarm. If an IP address is not specified, the 
Docker ascertains if the system has a single IP address and, 

if it does, the IP address and port 2337 is used. If the system 
has multiple IP addresses, the --advertise-addr must be 
specified for inter-manager communication and overlay 
networking. 


Availability of the node. Should be one of active 
active/pause/drain. 


Whether to force create a new cluster from the current state. false 
We discuss why it may be required to force create and use the 
option in this chapter. 


Listen address in the format «ip| interface»[:port ]. 0.0.0.0:2377 


Use the default values for all options except the - -advertise-addr for which a default value is not 
provided. Use the private address for the advertised address, which may be obtained from the EC2 console, 
as shown in Figure 2-5. If the EC2 instances on AWS were in different regions, the external public IP address 
should be used to access the manager node, which may also be obtained from the EC2 console. 


Launch Instance Connect Actions 'v 
E oie o 8 © 


©. Instance State : Running Add filter o 1to030f3 
Name -* Instance ID à Instance Type ~ Availability Zone ~ Instance State = ‘Status Checks ~ — Alarm Status Public DNS (IPvá) 

B SwamManager iD1b12315cb7cB33be — t2. micro us-east- Tf Q running & 2/2 checks Nane Me  0c2-34-204-168-217 co 
SwarmWorker i-05333632781259885 t2 micre us-east-1f @ running © N? checks Mane “jp ec 2-34-204-199-45. co 
SwanrmWorker i-08843beee6ed430d t2 micro us-east-1f @ running © 2 checks Nana Me 00c2-34-231-70-10. com 

Instance: | i-01b12215cb7c823be (SwarmManager) Public ONS: ec2-24-204-168-217.compute-1.amazonaws.com BEE - 

Description Status Checks Monitoring Tags Usage Instructions 
Instance ID — i-01b12315cb7c 833be Publie DNS (IPv4) — ec2-34-204-168-217 compute- 
l.amazonaws.com 
Instance stale — running [Pvt Public IP 34 204.168 217 
Instance type — L2 micro Pi IPs 
Elastic IPs Private DNS — ip-172-30-5-70.8c2 intemal 
Availabilty zone — us-east- Tf s Private IPs 172.90.5.70 
Security groups Container Linux by CoreOS -Stable—- 1409-7 Secondary private IPs 
Ü-AutogenBy AWSMP-1. view inbound rules 
Scheduled events No scheduled events VPCID 3 vpe-decÜüb5ba5 


Figure 2-5. Private IP 


Run the following command to initialize Docker Swarm mode. 


docker swarm init --advertise-addr 172.30.5.70 
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As the output in the following listing indicates, Swarm is initialized and the current node is a manager 
node. The command to add a worker node is also included in the output. The command to obtain the 
command to add a manager node is also output. Copy the docker swarm join command to add a worker 
node to the Swarm. 


core@ip-172-30-5-70 ~ $ docker swarm init --advertise-addr 172.30.5.70 
Swarm initialized: current node (bgzqx2cfsfO5qdradxytmdcp3) is now a manager. 


To add a worker to this swarm, run the following command: 
docker swarm join \ 
--token SWMTKN-1-303ziirxgkzy5gq5itr580yp9pbagxnkelinzh420vrb7znt6f- 
dmgeg3veppor942vsavma3s47 \ 
172.30.5.70:2377 
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the 
instructions. 


Run the docker info command to get system-wide information about the Docker Engine. The 
command outputs the total number of Docker containers that are running, paused, or stopped; partial 
output is listed. 


core@ip-172-30-5-70 ~ $ docker info 
Containers: O 


Running: O 
Paused: O 
Stopped: O 
Images: O 


Server Version: 1.12.6 

Storage Driver: overlay 

Backing Filesystem: extfs 

Logging Driver: json-file 
Cgroup Driver: cgroupfs 

Plugins: 

Volume: local 

Network: null host bridge overlay 
Swarm: active 

NodeID: bgzqx2cfsfo5qdradxytmdcp3 
Is Manager: true 

ClusterID: O56zmo5kk6em6u7vlki8pbhc9 
Managers: 1 

Nodes: 1 
CPUs: 1 
Total Memory: 994.6 MiB 

Name: ip-172-30-5-70.ec2.internal 
Docker Root Dir: /var/lib/docker 


The Storage Driver is overlay and the backing filesystem is extfs. The logging driver is json- file, 
which is covered in Chapter 11 on logging. The Swarm is shown to be active. Information about the node 
such as NodeID, whether the node is a manager, the number of managers in the Swarm, and the number of 
nodes in the Swarm, is also listed. 
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The resource capacity (CPU and memory) of the node is also listed. Chapter 7 discusses more about 
resource usage. The node name is the private DNS of the EC2 instance on which the Swarm is initialized. 
List the nodes in the Swarm with the following command: 


docker node ls 


A single node gets listed including the node ID, which is the only unique parameter for a node. 
The hostname is also unique if a node has not been made to leave the Swarm and rejoined. 


core@ip-172-30-5-70 ~ $ docker node ls 
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
bgzqx2cfsfO5qdradxytmdcp3 * ip-172-30-5-70.ec2.internal Ready Active Leader 


The * after the node ID indicates that this is the current node. The nodes in the Swarm also have a 
STATUS, AVAILABILITY, and MANAGER STATUS columns. STATUS can be one of the values listed in Table 2-2. 


Table 2-2. Node Status 


Status Description 
Ready Ready for use 
Down Not ready for use 
Unknown Not known 


AVAILABILITY can be one of the values listed in Table 2-3. 


Table 2-3. AVAILABILITY Column 


Availability Description 


Active Scheduler may assign tasks to the node. 
Pause Scheduler does not assign new tasks to the node but existing tasks keep running. 
Drain Scheduler does not assign new tasks to the node and existing tasks are shut down. 


Replacement tasks are started on other nodes. 


MANAGER STATUS can be one of the values listed in Table 2-4. If the MANAGER STATUS column has no 
value, it indicates a worker node. 
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Table 2-4. Manager Status 
Manager Status Description 


Reachable The node participates in the Raft consensus quorum and, if the leader becomes 
unavailable, the node is eligible to be made the leader node. 


Unreachable The node was a manager node that was reachable but has become unreachable 
and is not able to communicate with the other manager nodes in the Swarm. An 
unreachable manager node could be made reachable though not guaranteed to be 
restored by doing one of the following: 


-Restart the machine 
-Restart the daemon 
If neither of the preceding restores a unreachable manager node, the following should 


be implemented. 
Demote and remove the failed node. 


docker node demote «NODE» and docker node rm «id-node» 


Add another manager node with docker swarm join. 
Or 
Promote a worker node to manager node with docker node promote 


Leader Primary manager node that performs all the Swarm management and orchestration. 


Joining Nodes to the Swarm 


Additional nodes, manager or worker, may be added or joined to the Swarm as required. By default, manager 
nodes are also worker nodes but not vice versa. The manager nodes are added for a different reason than 

the worker nodes. The manager nodes are added to make the Swarm more fault tolerant and the worker 
nodes are added to add capacity to the Swarm. The commands to add manager and worker nodes are also 
different. The command to add a worker node is output when the Swarm is initialized. The command to add 
a worker node may also be found using the following command. 


docker swarm join-token worker 

The command to add a manager node may be found using the following command. 
docker swarm join-token manager 

A reason for adding a worker node is that the service tasks scheduled on some of the nodes are not 
running and are in Allocated state. A reason for adding a manager node is that another manager node has 
become unreachable. 

The node to join, manager or worker, must have Docker Engine version at least 1.12 installed. Next, you 
add two worker nodes. Obtain the public IP address of an EC2 instance started for a worker node. SSH login 


to the worker instance. 


ssh -i "coreos.pem" core@34.204.199. 
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Run the docker swarm join command, which has the following syntax, to join the node to the Swarm 
as a worker node. 


docker swarm join [OPTIONS] HOST:PORT 


The options supported by the docker swarm join command are listed in Table 2-5. 


Table 2-5. Options for docker swarm join Command 


Option Description Default Value 
--advertise-addr Advertised address in format «ip | interface»[:port ]. 

--availability Availability of the node. One of active/pause/drain. active 
--listen-addr Listen address in format <ip|interface>[:port]. 0.0.0.0:2377 
- -token Token for entry into the Swarm. 


Run the docker swarm join command output during the initialization of the Swarm mode to join the 
worker instance with the Swarm. As the output message indicates, "The node joined the Swarm as a worker.’ 


[root@localhost ~]# ssh -i "coreos.pem" core@34.204.199.45 

Container Linux by CoreOS stable (1409.7.0) 

core@ip-172-30-5-31 ~ $ docker swarm join \ 

> --token SWMTKN-1-303ziirxgkzy5gq5itr580yp9pbagxnkelinzh420vrb7znt6f- 
dmgeg3veppor942vsavma3s47 \ 

> 172.30.5.70:2377 

This node joined a swarm as a worker. 


Similarly, SSH login to the other worker instance. 
ssh -i "coreos.pem" core@34.231.70.10 
Run the same docker swarm join command and the second nodes joins the Swarm as a worker node. 


[root@localhost ~]# ssh -i "coreos.pem" core@34.231.70.10 

Container Linux by CoreOS stable (1409.7.0) 

core@ip-172-30-5-108 ~ $ docker swarm join \ 

> --token SWMTKN-1-303ziirxgkzy5gq5itr580yp9pbagxnkelinzh420vrb7znt6f- 
dmgeg3veppor942vsavma3s47 \ 

> 172.30.5.70:2377 

This node joined a swarm as a worker. 


The following sequence of events takes place when the docker swarm join command runs to join a 
worker node to the Swarm. 
1. TheSwarm mode for the Docker Engine on the node is enabled. 
A request for a TLS certificate is sent to the manager. 


2 
3. The node is named with the machine hostname. 
4 


The current node joins the Swarm at the manager listen address. Based on the 
token, the node is joined as a worker node or a manager node. 
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5. Sets the current node to Active availability. 
6. The ingress overlay network is extended to the current node. 


When a node is joined to the Swarm using the manager token, the node joins as a manager node. 

The new manager nodes should be Reachable and only the first manager node is the leader. Leader election 
to a different manager node occurs only if the initial leader node were to fail or be demoted. 

The worker nodes differ from the manager nodes in another regard. A worker node cannot be used to 
view or modify the cluster state. Only the manager node can be used to view the cluster state such as the 
nodes in the Swarm. Only the manager node can be used to modify a cluster state such as remove a node. 

If the docker node 1s command is run on a worker node, the following error message is generated. 


core@ip-172-30-5-31 ~ $ docker node ls 
Error response from daemon: This node is not a swarm manager. Worker nodes can't be used 


to view or modify cluster state. Please run this command on a manager node or promote the 
current node to a manager. 


Testing the Swarm 


Next, you deploy a simple Hello World service to the Swarm to test the cluster. List the nodes in the Swarm 
from the manager node with the following command. 


docker node ls 
The three nodes should be listed. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
9n5qmj4pp91fOn3sOn2jwjdv8 ip-172-30-5-108.ec2.internal Ready Active 
bgzqx2cfsfo5qdradxytmdcp3 * ip-172-30-5-70.ec2.internal Ready Active Leader 


bqq4bryuobyluOglm4pi9tko4 ^ ip-172-30-5-31.ec2.internal Ready Active 


How do you tell if a node is a manager node or a worker node? From the Manager Status column. If the 
Manager Status is empty, the node is a worker node and if the Manager Status has a value, which would be 
one of the values discussed in Table 2-4, the node is a manager node. Two worker nodes and one manager 
node are listed. 

We already discussed that worker nodes can't be used to view or modify cluster state. Next, create a 
Docker service using the docker service create command, which becomes available only if the Swarm 
mode is enabled. Using Docker image alpine, which is a Linux distribution, create two replicas and ping the 
docker.com domain from the service containers. 


docker service create --replicas 2 --name helloworld alpine ping docker.com 


If the preceding command runs without an error, the Docker Swarm installed fine. The command 
returns the service ID. 


core@ip-172-30-5-70 ~ $ docker service create --replicas 2 --name helloworld alpine ping 


docker.com 
bkwskfzqa173dp55j54erg5cg 
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Services may be listed with the following command. 
docker service ls 


The service helloworld is listed and the number of replicas is listed as 2/2, which implies that two 
replicas exist and meet the desired state of two replicas. The REPLICAS column output is ordered “actual/ 
desired” The Docker image is alpine and the command to run the service is ping docker.com. 


core@ip-172-30-5-70 ~ $ docker service ls 
ID NAME REPLICAS IMAGE COMMAND 
bkwskfzqa173 helloworld 2/2 alpine ping docker.com 


The docker service inspect command is used to find more information about the service. 
docker service inspect helloworld 


The detailed information about the helloworld service—including the container spec, resources, 
restart policy, placement, mode, update config, and update status—is listed. 


core@ip-172-30-5-70 ~ $ docker service inspect helloworld 
[ 
{ 
"ID": "bkwskfzqa173dp55j54erg5cg", 


"Version": { 
Index": 22 


"CreatedAt": "2017-07-227T19:11:50. 3458234662", 
"UpdatedAt": "2017-07-22T19:11:50. 3458234662", 
"Spec": { 
"Name": "helloworld", 
"TaskTemplate": 1 
"ContainerSpec": { 
"Image": "alpine", 


"Args": [ 
"ping", 
"docker. com" 
] 
Jo 
"Resources": { 
"Limits": {}, 
"Reservations": {} 
) 


"RestartPolicy": { 
"Condition": "any", 
"MaxAttempts": O 


Jo 
"Placement": {} 
)» 
"Mode": ( 
"Replicated": { 
"Replicas": 2 
} 
Jo 


21 


CHAPTER 2. USING DOCKER IN SWARM MODE 


"UpdateConfig": { 
"Parallelism": 1, 


"FailureAction": "pause" 
Js 
"EndpointSpec": { 
"Mode": "vip" 
j 
Js 
"Endpoint": { 
"Spec": {} 
Jy 


"UpdateStatus": { 
"StartedAt": "0001-01-01T00:00:00Z", 
"CompletedAt": "0001-01-01T00:00:00Z" 


The replicas and the nodes on which the replicas are placed may be listed with the following command 
syntax. 


docker service ps «SERVICE 


The «SERVICE» placeholder is either a service name (like helloworld)or the actual service ID 
(like bkwskfzqa173 for this example). For the helloworld service, the command becomes: 


docker service ps helloworld 


The preceding command also lists the node on which a replica is running. The Docker containers 
started for a service are listed with same command as before, the docker ps command. 


core@ip-172-30-5-70 ~ $ docker service ps helloworld 

ID NAME IMAGE NODE DESIRED STATE 
CURRENT STATE ERROR 

2x8gqd2qbylpkugikgOpxiic2 helloworld.1 alpine ip-172-30-5-70.ec2.internal Running 
Running 34 seconds ago 

6twqivOlr2gfilnb6éae19hrpx9 helloworld.2 alpine ip-172-30-5-108.ec2.internal Running 
Running 34 seconds ago 

core@ip-172-30-5-70 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED STATUS 
PORTS NAMES 
acbdaccad6ea alpine:latest "ping docker.com" 47 seconds ago Up 46 seconds 


helloworld.1.2x8gqd2qbylpkugikgopxi1c2 
The docker ps command is not a Swarm mode command, but may be run on the worker nodes to find 


the service containers running on a worker node. The docker ps command gives you all containers running 
on a node, even if they are not service containers. 
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core@ip-172-30-5-108 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED STATUS 
PORTS NAMES 
74ea31054fb4 alpine:latest "ping docker.com" About a minute ago Up About a minute 


helloworld.2.6twqivolr2gflnb6aei9hrpx9 


Only two nodes are listed by the docker service ps helloworld command on which replicas are 
scheduled, the manager node and one of the worker nodes. The docker ps command on the other worker 
node does not list any Docker containers. 


coreQip-172-30-5-31 ~ $ docker ps 
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 


The docker node inspect <node> command is used to get detailed information about a node, such as 
the node role, availability, hostname, resources capacity, plugins, and status. 


core@ip-172-30-5-70 ~ $ docker node inspect ip-172-30-5-70.ec2.internal 


{ 
"ID": “bgzqx2cfsfo5qdradxytmdcp3", 
"Version": { 

Index": 10 


"CreatedAt": "2017-07-22T19:09:45.647701768Z" , 
"UpdatedAt": "2017-07-22T19:09:45.68030039Z", 
"Spec": { 

"Role": "manager", 

"Availability": "active" 
Jy 
"Description": { 

"Hostname": "ip-172-30-5-70.ec2.internal", 


"Platform": { 
"Architecture": "x86 64", 
"OS": "linux" 

) 


"Resources": ( 
"NanoCPUs": 1000000000, 
"MemoryBytes": 1042935808 


Js 
"Engine": { 
"EngineVersion": "1.12.6", 
"Plugins": [ 
{ 


"Type": "Network", 
"Name": "bridge" 


Jo 

{ 
"Type": "Network", 
"Name": "host" 

Jo 
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{ 
"Type": "Network", 
"Name": "null" 
Jo 
{ 
"Type": "Network", 
"Name": "overlay" 
Jo 
{ 
"Type": "Volume", 
"Name": "local" 
} 
] 
} 
Jo 
"Status": { 
"State": "ready" 
)» 


"ManagerStatus": { 
"Leader": true, 
"Reachability": "reachable", 
"Addr": "172.30.5.70:2377" 


A service may be removed with the docker service rm «service» command. Subsequently, the 
docker service inspect «service» command should not list any replicas and running docker ps will 
show no more running Docker containers. 


coreQip-172-30-5-70 ~ $ docker service rm helloworld 
helloworld 
core@ip-172-30-5-70 ~ $ docker service inspect helloworld 


[] 


Error: no such service: helloworld 


Chapter 4 discusses more about services. 


Promoting a Worker Node to Manager 


As mentioned before, a manager node is also a worker node by default, but a worker node is only a worker 
node. But a worker node may be promoted to a manager node. The Docker command to promote one or 
more worker nodes to a manager node has the following syntax. 


docker node promote NODE [NODE...] 
The command must be run from the leader node. As an example, promote the node ip-172-30-5-108. 


ec2.internal. As the output indicates, the node gets promoted to a manager node. Subsequently list the 
nodes in the Swarm and the node promoted should have manager status as Reachable. 
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A worker node should preferably be promoted using the node ID; the reason for which is discussed 
subsequently. Promote another worker node using the node ID. Subsequently, both the worker nodes are 
listed as Reachable in the Manager Status column. 


core@ip-172-30-5-70 ~ $ docker node promote ip-172-30-5-108.ec2.internal 
Node ip-172-30-5-108.ec2.internal promoted to a manager in the swarm. 
core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
9n5qmj4pp91f0n3s0n2jwjdv8 ip-172-30-5-108.ec2.internal Ready Active Reachable 
bgzqx2cfsfO5qdradxytmdcp3 * ip-172-30-5-70.ec2.internal Ready Active Leader 


bqq4bryuobyluoglm4pi9tko4 ip-172-30-5-31.ec2.internal Ready Active 


Demoting a Manager Node to Worker 


A manager node may be demoted to a worker node with the following Docker command. 
docker node demote NODE [NODE...] 


Any manager node, including the leader node, may be demoted. As an example, demote the manager 
node ip-172-30-5-108.ec2. internal. 


core@ip-172-30-5-70 ~“ $ docker node demote  ip-172-30-5-108.ec2.internal 
Manager ip-172-30-5-108.ec2.internal demoted in the swarm. 


Once demoted, the commands such as docker node ls that can be run only from a manager node 
cannot be run any more on the node. The docker node 1s command lists the demoted node as a worker 


node; no MANAGER STATUS is listed for a worker node. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
9n5qmj4pp91f0n3s0n2jwjdv8 ip-172-30-5-108.ec2.internal Ready Active 
bgzqx2cfsfO5qdradxytmdcp3 * ip-172-30-5-70.ec2.internal Ready Active Leader 


bqq4bryuobyluoglm4pi9tko4 ip-172-30-5-31.ec2.internal Ready Active 


A node should be preferably promoted/demoted and otherwise referred to in any command that is 
directed at the node using the node ID, which is unique to a node. The reason being that a demoted node, if 
promoted back, could be added with a different node ID and the docker node ls command could list two 
node IDs for the same hostname. If the hostname is used to refer to a node, it could result in the node is 
ambiguous error message. 


Making a Worker Node Leave the Swarm 


Earlier you joined a node to the Swarm as a worker node. A worker node may also be made to leave the 
Swarm. As an example, make one of the worker nodes leave with the following command, which must be 
run from the node you want to remove from the Swarm. 


docker swarm leave 
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As the message output indicates, the node has left the Swarm. 


core@ip-172-30-5-31 ~ $ docker swarm leave 
Node left the swarm. 


Similarly, make the other worker node leave the Swarm. 


core@ip-172-30-5-108 ~ $ docker swarm leave 
Node left the swarm. 


After a worker node has left the Swarm, the node itself is not removed and continues to be listed with 
the docker node 1s command with a Down status. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
9n5qmj4pp91f0n3s0n2jwjdv8 ip-172-30-5-108.ec2.internal Down Active 
bgzqx2cfsfo5qdradxytmdcp3 * ip-172-30-5-70.ec2.internal Ready Active Leader 


bqq4bryuobyludg1m4p19tko4 ip-172-30-5-31.ec2.internal Down Active 


Making a Manager Node Leave the Swarm 


While it is easier to make a worker node leave the Swarm, it is different when a manager node must leave the 
Swarm. Making a worker node leave the Swarm only lowers the Swarm capacity in terms of the service tasks 
that may be scheduled in the Swarm. But making a manager node leave the Swarm makes the Swarm less 
available. If the fault tolerance does not allow for a manager node to fail or be removed from the Swarm, the 
same docker swarm leave command that made a worker node leave the Swarm cannot be used to make a 
manager node leave the Swarm. If a Swarm has only one manager node, the docker swarm leave command 
generates the following error message. 


core@ip-172-30-5-70 ~ $ docker swarm leave 

Error response from daemon: You are attempting to leave the swarm on a node that is 
participating as a manager. Removing the last manager erases all current state of the 
Swarm. Use `--force to ignore this message. 


Add the - - force option to the docker swarm leave command on the manager node to cause the 
manager node to leave the Swarm. 


core@ip-172-30-5-70 ~ $ docker swarm leave --force 
Node left the swarm. 


If the only manager node is removed, the Swarm no longer exists. The Swarm must be initialized again 
if the Swarm mode is to be used. 


core@ip-172-30-5-70 ~ $ docker swarm init --advertise-addr 172.30.5.70 
Swarm initialized: current node (cnyc2w3n8q8zuxjujcd2s729k) is now a manager. 
To add a worker to this swarm, run the following command: 
docker swarm join \ 
--token SWMTKN-1-4lxmisvlszjgck4lyOswsxubejfxophlneixegho2fiq99amqf- 
11mpscd8gs6bsayzren8fa2ki \ 
172.30.5.70:2377 
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the 
instructions. 
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A new Swarm is created with only the manager node and the Swarm has only one node initially. 


core@ip-172-30-5-70 ~ $ docker node ls 
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
cnyc2w3n8q8zuxjujcd2s729k * ip-172-30-5-70.ec2.internal Ready Active Leader 


If a Swarm has two manager nodes, making one of the manager nodes leave the Swarm has a different 
effect. With two managers, the fault tolerance is 0, as discussed earlier. To create a Swarm with two manager 


nodes, start with a Swarm that has one manager node and two worker nodes. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
4z03hudbo3fz17q94leo24pvh ip-172-30-5-108.ec2.internal Ready Active 
cnyc2w3n8q8zuxjujcd2s729k * ip-172-30-5-70.ec2.internal Ready Active Leader 


efsxwt43iskasa6poh2stkjeb ip-172-30-5-31.ec2.internal Ready Active 
Promote one of the worker nodes to a manager node. 


core@ip-172-30-5-70 ~ $ docker node promote ip-172-30-5-108.ec2.internal 
Node ip-172-30-5-108.ec2.internal promoted to a manager in the swarm. 


The Swarm will then have two manager nodes. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
4z03hudbo3fz17q94leo24pvh ip-172-30-5-108.ec2.internal Ready Active Reachable 
cnyc2w3n8q8zuxjujcd2s729k * ip-172-30-5-70.ec2.internal Ready Active Leader 


efsxwt43iskasa6poh2stkjeb ip-172-30-5-31.ec2.internal Ready Active 


Run the docker swarm leave command from a manager node that’s not the leader node. The following 
message is generated. 


core@ip-172-30-5-108 ~ $ docker swarm leave 
The error response from the daemon is as follows: 
You are attempting to leave the swarm on a node that is participating as a manager. 
Removing this node leaves one manager out of two. Without a Raft quorum, your Swarm will be 
inaccessible. The only way to restore a Swarm that has lost consensus is to reinitialize it with - - force-new- 
cluster. Use --force to suppress this message. 


To make the manager node leave, you must add the - - force option to the command. 


core@ip-172-30-5-108 ~ $ docker swarm leave --force 
Node left the swarm. 
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When one of the two managers has left the Swarm, the Raft quorum is lost and the Swarm becomes 
inaccessible. As indicated, the Swarm must be reinitialized using the - - force-new-cluster option. 


Reinitializing a Cluster 


A Swarm that has lost quorum cannot be reinitialized using the command used to initialize a Swarm. If the 
same command runs on a Swarm that has lost quorum, a message indicates that the node is already in the 
Swarm and first must be made to leave the Swarm: 


coreQip-172-30-5-70 ~ $ docker swarm init --advertise-addr 172.30.5.70 
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" 
to leave this swarm and join another one. 


To reinitialize the Swarm the --force-new-cluster option must be added to the docker swarm 
init command. core@ip-172-30-5-70 ^ $ docker swarm init --advertise-addr 172.30.5.70 
- -force-new-cluster 
Swarm initialized: current node (cnyc2w3n8q8zuxjujcd2s729k) is now a manager. 
To add a worker to this swarm, run the following command: 
docker swarm join \ 
--token SWMTKN-1-4lxmisvlszjgck4lyOswsxubejfxophlneixegho2fiq99amqf- 
11mpscd8gs6bsayzren8fa2ki \ 
172.30.5.70:2377 
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the 
instructions. 


The Swarm is reinitialized and the docker swarm join command to add a worker node is output. 


Modifying Node Availability 


The availability of a node may be modified with the D command with the -- availability option. One of 
the - -availability options shown in Table 2-6 may be set. 


Table 2-6. Availability Options 


Availability Option Description 


active Restores a paused or drained node to active. 
pause Pauses a node so that itis not available to receive new tasks. 
drain With a worker node, the node becomes down and unavailable for scheduling new 


tasks. A manager node also becomes unavailable for scheduling new tasks but 
continues to perform Swarm management. 


As an example, you can drain a worker node as follows. 


core@ip-172-30-5-70 ~ $ docker node update --availability drain ip-172-30-5-108.ec2.internal 
ip-172-30-5-108.ec2.internal 
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The worker node is drained. All service tasks on the drained node are shut down and started on other 
nodes that are available. The output from the docker node ls command lists the node with the status set to 
Drain. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
bhuzgyqvb83dx0zvms5400a58 ip-172-30-5-108.ec2.internal Ready Drain 
cnyc2w3n8q8zuxjujcd2s729k * ip-172-30-5-70.ec2.internal Ready Active Leader 


efsxwt43iskasa6poh2stkjeb ip-172-30-5-31.ec2.internal Ready Active 


The node detail (partial output is listed) for the drained worker node lists the node 
availability as "drain".coreQip-172-30-5-70 ~“ $ docker node inspect ip-172-30-5-108.ec2. 
internal 
l 
{ 
"ID": "bhuzgyqvb83dxOzvms5400a58", 
"Version": { 
Index": 49 


"CreatedAt": "2017-07-22T19:30:31.544403951Z", 
"UpdatedAt": "2017-07-22T19:33:37.45659544Z", 
"Spec": { 
"Role": "worker", 
"Availability": "drain" 
Jy 
"Description": { 
"Hostname": "ip-172-30-5-108.ec2.internal", 
All service tasks on the drained node are shut down and started on other nodes that are available. 
The node availability with the docker node ls is listed as Drain. 
A drained node can be made active again using the docker node update command with 


--availability set to Active. 


core@ip-172-30-5-70 ~ $ docker node update --availability active ip-172-30-5-108.ec2.internal 
ip-172-30-5-108.ec2. internal 


The drained node becomes active and is listed with the status set to Active. 


core@ip-172-30-5-70 ~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
bhuzgyqvb83dx0zvms5400a58 ip-172-30-5-108.ec2.internal Ready Active 
cnyc2w3n8q8zuxjujcd2s729k * ip-172-30-5-70.ec2.internal Ready Active Leader 


efsxwt43iskasa6poh2stkjeb ip-172-30-5-31.ec2.internal Ready Active 
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Removing a Node 


One or more nodes may be removed from the Swarm using the docker node rm command, which is run 
from any manager node. 


docker node rm [OPTIONS] NODE [NODE...] 


The difference between docker swarm leave and docker node rmis thatthe docker node rm may be run 
only from a manager node. A demoted node can only be removed from the Swarm with the docker node rm 
command. The sequence to remove a manager node without using the - - force option is the following. 


1. Demote the manager node, which makes it a worker node. 
2. Drain the worker node. 

3. Make the worker node leave the Swarm. 

4 


Remove the node. 


Summary 


This chapter discussed using Docker in Swarm mode. First, you initialized the Swarm mode with the docker 
swarm init command to make the current node the manager node in the Swarm. Subsequently, you joined 
worker nodes to the Swarm with the docker swarm join command. The chapter also discussed promoting 
a worker node to a manager node/demoting a manager node to a worker node, making a worker node leave 
a Swarm and then rejoin the Swarm, making a manager node leave a Swarm, reinitializing a Swarm, and 
modifying node availability and removing a node. The next chapter introduces Docker for AWS, which is a 
managed service for Docker Swarm mode. 
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CHAPTER 3 


Using Docker for AWS to Create a 
Multi-Zone Swarm 





Docker Swarm is provisioned by first initiating a Swarm to create a manager node and subsequently joining 
worker nodes to that manager node. Docker Swarm provides distributed service deployment for Docker 
applications. 


The Problem 


By default, a Docker Swarm is provisioned on a single zone on AWS, as illustrated in Figure 3-1. With the 
manager nodes and all the worker nodes in the same AWS zone, failure of the zone would make the zone 
unavailable. A single-zone Swarm is not a highly available Swarm and has no fault tolerance. 


Swarm Manager 
Node 





Swarm Worker Node Swarm Worker Node Swarm Worker Node 


Figure 3-1. A single-zone Swarm 
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The Solution 


Docker and AWS have partnered to create a Docker for AWS deployment platform that provisions a Docker 
Swarm across multiple zones on AWS. Docker for AWS does not require users to run any commands on a 
command line and is graphical user interface (GUI) based. With manager and worker nodes in multiple 
zones, failure of a single AWS zone does not make the Swarm unavailable, as illustrated in Figure 3-2. Docker 
for AWS provides fault tolerance to a Swarm. 


Docker 
Swarm 





>< i 


Swarm 
Manager 


Swarm Swarm 


Manager Manager 





Figure 3-2. A Multi-zone Swarm 
Docker for AWS is a managed service for Docker Swarm on the AWS cloud platform. In addition to 
multiple zones, Docker for AWS has several other benefits: 
e  Allthe required infrastructure is provisioned automatically. 
e Automatic upgrade to new software versions without service interruption. 


e  Acustom Linux distribution optimized for Docker. The custom Linux distribution is 
not available separately on AWS and uses the overlay2 storage driver. 


e Unused Docker resources are pruned automatically. 


e — Auto-scaling groups for managing nodes. 


32 


CHAPTER 3 ~ USING DOCKER FOR AWS TO CREATE A MULTI-ZONE SWARM 


e Log rotation native to the host to avoid chatty logs consuming all the disk space. 
e Centralized logging with AWS CloudWatch. 
e A bug-reporting tool based on a docker-diagnose script. 
Two editions of Docker for Swarm are available: 
e Docker Enterprise Edition (EE) for AWS 
e Docker Community Edition (CE) for AWS 


We use the Docker Community Edition (CE) for AWS in this chapter to create a multi-zone Swarm. 
This chapter includes the following topics: 


e Setting the environment 

e Creating a AWS CloudFormation stack for the Docker Swarm 
e Connecting with the Swarm manager 

e Using the Swarm 


e Deleting the Swarm 


Setting the Environment 


Two deployment options are available with Docker for AWS. 
e Useapre-existing VPC 
e Use anew VPC created by Docker 


Letting Docker create the VPC, subnets, and gateways is the easier option and the one used in this 
chapter. 

Create an AWS account if you don’t already have one at https: //aws.amazon.com/resources/create- 
account/. The AWS account must support EC2-VPC. Even though AWS services such as VPC are created 
automatically, the account must have permissions to create EC2 instances, including auto-scaling groups, 
IAM profiles, DynamoDB tables, SQS Queue, VPC (including subnets, gateways, and security groups), Elastic 
Load Balancer, and CloudWatch Log Group. The only user input other than creating an account with the 
required permissions is to create a SSH key pair in the AWS region in the Docker Swarm. 

Select the EC2 AWS service and click on the Key Pairs link in the EC2 dashboard. Click on Create Key Pair 
to create and download a key pair. Specify a key pair name (docker for example) in the Create Key Pair dialog 
and click on Create. A key pair gets created, as shown in Figure 3-3. Copy the key pair file (docker.pem) to a 
local Linux machine. 


Services v Resource Groups v '* Ó Deepak Vohra» Ohio~ Support v 





Elastic s SALGA Import Key Pair Delete 
East Ps ! port Key o + © 


Placement Groups 


| Key Pairs Q € 1to20f2 
Network Interfaces : ; 
Key pair name ^ Fingerprint 
" PAK coreos 5f:18:ce:08:2b:2e:a9:d2: 7f: 7a: a5: 6c 0e: 39:07: e7: 74:18: 72:31 
Load Balancers 
@ docker 3e.a5:ea:08:30.40.b3:73:04:7¢-15:a0:35:2¢:d6:d4:4e:41:0¢:44 


Target Groups L 
g 


Figure 3-3. A key pair 
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Set the permissions on the docker. pem to 400, which gives only read permissions and removes all other 
permissions. 


chmod 400 docker.pem 


Creating a AWS CloudFormation Stack for Docker Swarm 


Navigate to https: //docs.docker.com/docker-for-aws/ in a web browser and click on the Deploy Docker 
for AWS option, as as shown in Figure 3-4. The label could be different, such as Deploy Docker Community 
Edition [CE] for AWS [stable]. 


Deploy Docker Deploy Docker Deploy Docker 
Community Edition Community Edition Community Edition 
[CE] for AWS q» [CE] for AWS [edge] [CE] for AWS [test] 
[stable] 
Deploy Docker Deploy Docker 


Deploy Docker Community Edition Community Edition 
Community Edition [CE] for AWS [edge] [CE] for AWS [test] 
[CE] for AWS uses your existing VPC uses your existing VPC 
[stable] 
uses your existing VPC 





Figure 3-4. Deploy Docker for AWS 


The Create Stack wizard is started with the provision to either design a new template or choose the 
default CloudFormation template for Docker on AWS. Select the Specify an Amazon S3 Template URL option 
for which a URL is pre-specified, as shown in Figure 3-5. Click on Next. 
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(D CloudFormation ~ Stacks > Create Stack 


Create stack 


| Select Template Select Template 

Specify Details 
Options Select the template that describes the stack that you want to create. A stack is a group of related resources that vou manage asa single unit 
Review 


Design a template Use AWS CloudFormation Designer to create or modify an existing template. Learn more 


Design template 


Choose a template A template is a JSON/YAML-formatied text file that describes your stack's resources and their properties. Learn more 


Select a sample template 


Upload a template to Amazon $3 
Choose File No ffe chosen 
® Speci^y an Amazon S3 template URL 


https //eddions-us-east-1.53.amazonaws.com/aws/stable/Docker.ti.| View/Edit template in Designer 


ES C 


Figure 3-5. Selecting a template 


In Specify Details, specify a stack name (DockerSwarm). The Swarm Parameters section has the fields 
listed in Table 3-1. 


Table 3-1. Swarm Parameters 


Parameter Description 
Number of Swarm managers? Number of Swarm manager nodes. Valid values are 1, 3, and 5. 
Number of Swarm worker nodes? Number of worker nodes in the Swarm (0-1000). 


Keep the default settings of 3 for Number of Swarm Managers and 5 for Number of Swarm Worker 
nodes, as shown in Figure 3-6. 
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Create stack 





select Template Specify Details 
| Specify Details 
Options Spec fy a stack name anc parameter values. You can use or change ihe cefault parameter values, which are defined in the AWS CloudFormation template Lear more 
Review 
Stack name DockerSwarm 
Parameters 
Swarm Size 
Number of Swarm j v | Number of Swarm manager nodes (* 
managers? 1 | 


Number of Swarm worker 
nodes? 


Swarm Properties 


Which SSH key to use? X 


Figure 3-6. Specifying a stack name 
Next, specify the Swarm properties, as discussed in Table 3-2. 


Table 3-2. Swarm Properties 
Swarm Property Description Value Set 


Which SSH key to use? Name of an existing EC2 key pair to enable docker 
SSH access to the instances. 


Enable daily resource cleanup? Cleans up unused images, containers, no 
networks, and volumes. 


Use CloudWatch for container logging? Send all container logs to CloudWatch. yes 


In the Which SSH key to use? property, select the docker SSH key. The Swarm properties are shown in 
Figure 3-7. 
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Parameters 


Swarm Size 


Number of Swarm 
managers? 


Number of Swarm worker 
nodes? 


Swarm Properties 


Which SSH key to use? 


Enable daily resource 
cleanup? 


Use Cloudwatch for 
container logging? 


Create EFS prerequsities 
for CloudStor? 
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3 vx | Number of Swarm manager nodes (1, 3, 5) 

5 Number of worker nodes in the Swarm (0-1000) 
= 

docker - 


A. 


Mame of an existing EC2 KeyPair to enable SSH access to the instances 


no T| Cleans up unused images, containers, networks and volumes 
ves T| Send all Container logs to CloudVWatch 
no vx | Create CloudStor EFS mount targets 


Swarm Manager Properties 


Figure 3-7. Swarm properties 


Specify the Swarm Manager properties, as discussed in Table 3-3. 


Table 3-3. Swarm Manager Properties 


Swarm Property Description Value Set 
Swarm manager instance type? EC2 HVM instance type (t2.micro, m3.medium, etc.) t2.micro 
Manager ephemeral storage Size of manager's ephemeral storage volume in GB 20 
volume size? 

Manager ephemeral storage Manager volume type standard 


volume type? 
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The Swarm Manager properties are as shown in Figure 3-8. Specify the Swarm Worker properties, as 
discussed in Table 3-4. 


Table 3-4. Swarm Worker Properties 


Swarm Worker Property Description Value Set 
Agent worker instance type? EC2HVM instance — t2.micro 
type (t2.micro, 


m3.medium, etc.) 


Worker ephemeral storage volume size? Size of worker's 20 
ephemeral storage 
volume in GB 


Worker ephemeral storage volume type? Worker volumetype standard 


The Swarm Worker properties are shown in Figure 3-8. Click on Next. 


Swarm Manager Properties 


Swarm manager instance = {2. micro v | EC2 HVM instance type (t2 
type? 


Manager ephemeral 20 Size of Manager's ephemeral storage volume in GiB 
storage volume size? 


Manager ephemeral standard v | Manager ephemeral storage volume type 
storage volume type 


Swarm Worker Properties 


Agent worker instance t2. micro v| EC2HVI ance t 2 
type? 
Worker ephemeral storage 20 Size of Workers's eohemeral storage volume in GiB 
volume size? 
Worker ephemeral storage Standard v | Worker ephemeral storage volume type 


volume type 


Cancel Previous 





Figure 3-8. Swarm worker properties 
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Next, specify the options for the stack. Tags (key-value pairs) may be specified for resources in a stack. 
For permissions, an IAM role for CloudFormation may be chosen. None of these options is required to be 
set, as shown in Figure 3-9. 


dt 
Select Template Options 
Specity Detail 
| Options Tags 
Me > 
You can specify lags (key-value pairs for resources in your Stack You can add up to 50 unique key-value pairs for each stack eam more 
Key Value 
Permissions 
ou can choose an IAM role that CloudFormation uses to create. modify. or delete resources in the stack. If you Gon t choose a role. CloudFormation uses the permissions 
defined ( account. Leam re 
IAM Role 
Enter role am 


> Advanced 


You can set additional options for your stack. like notification options and a stack policy. Leam more 


Figure 3-9. Optional settings 


For Advanced options, the Notification options are set to No Notification. Set Rollback on Failure to 
Yes, as shown in Figure 3-10. Click on Next. 
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v Advanced 


You can set additional options for your stack, like notification options and a stack policy. Learn more. 


Notification options 


e No notification 


New Amazon SNS topic 


Topic 


Email 


Existing Amazon SNS topic 


Existing topic ARN 


Timeout @ Minutes 


Rollback on failure @ © Yes 


I? @No 


Stack policy @ © Enter policy 


Figure 3-10. Setting rollback on failure 
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Review the stack settings, as shown in Figure 3-11. 


(D CloudFormation v Stacks > Create Stack 


Create stack 


Select Template Review 
Specify Details 
Options Template 
| Review 
Template URL hitps://editions-us-east-1.s3.amazonaws.com/aws/stable/Docker tmp! 
Description Docker CE for AWS 17.06.0-ce (17.06.0-ce-aws2) 
Estimate cost Link is not available 
Details 
Stackname: DockerSwarm 
Swarm Size 


ManagerSize 3 
ClusterSize 5 


Swarm Properties 


KeyName docker 
EnableSystemPrune no 
EnableCloudWatchLogs yes 
EnableCloudStorEfs no 


Swarm Manager Properties 


Figure 3-11. Reviewing the stack settings 
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Select the acknowledgement checkbox and then click on Create, as shown in Figure 3-12. 


Options 


Tags 
No tags provided 
Advanced 

Notification 


Timeout none 
Rollback on failure — Yes 


Capabilities 


e» The following resource(s) require capabilities: [AWS::LAM::Role] 


This template contains Identity and Access Management (IAM) resources that might provide entities access to make changes to your AWS account. Check 
that you want to create each of these resources and that they have the minimum required permissions. Learn more. 





* | acknowledge that AWS CloudFormation might create IAM resources. 


Cancel Previous 





Figure 3-12. Creating the stack 


A new stack begins to be created. Click on the Refresh button to refresh the stacks listed, as shown in 
Figure 3-13. 


(D CloudFormation ~ Stacks 





dd (Actions ~ Design template c o 


Filter: Active * Showif Retresh KS 
Create a stack 
AWS CloudFormation allows you to quickly and easily deploy your infrastructure resources 
and applications on AWS. You can use one of the templates we provide to get started quickly 


with applic ations like WordPress or Drupal, one of tne many sample templates or create your 
own template. 


You da not cu meniy nave any Slacks. Choose Create new stack below to create a new AVIS 
CloudFormation stack 





Design a template 


Templates tell AWS CloudFonmnabion which AWS resources 10 provision and how to provision 
them. When you create a CloudPormation stack, you must submit a template 


Ta build and view templates. you can use tfe drag-and-drop tool Called AWS CiogdFormalicn 


Petite Nana iir dnd nb non ha rea sr iiec Ph wnn want ba Ael ha weer Firnnlalé nd deo 


Figure 3-13. Refresh 
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A new stack based on a CloudFormation template for Docker Swarm starts to be created, as indicated 
by the status CREATE IN PROGRESS shown in Figure 3-14. 


mq CloudFormation ~ Stacks 





Acbons + Design template g |- 
Filter: Acie + Showing 1 siack 
Stack Name Created Time Status Description 
Got herSwarm 2017-07-22 14:89:31 LITC-DTUO CREATE IN PROGRESS Docker CE for ANS 17 06 0-ce (17.05. 0-ce-aws d] 


Figure 3-14. CloudFormation stack status 


The different tabs are provided for the different stack details. The Resources tab shows the AWS 
resources created by the CloudFormation template, as shown in Figure 3-15. 


@ CloudFormation ~ Stacks 





Create Stack Mi Actions Design template C @ 
Filter: Active + Showing 1 sack 
Stack Name Created Time Status Description 
7 Dockerswanm 2017-07-22 14:29:31 UTC-0700 CREATE IM PROGRESS Docker CE for AWS 17.06.0-ce (17.06. 0-ce-aws2) 
Overview Outputs Resources Events Template Parameters Tags StackPolicy Change Sets BFE 
Logical ID Physical ID Type Status Status Reason 
2017/7 /22/]8LATEST|B58BIGSBa72 T de 3795432182 
AZinfo an) 22/8LATEST]|8588f668a7274ebbb15790432 19a Custom-AZinfo CREATE COMPLETE 
AzInfoFunction Dockerzwarm-Azinforunction- ELF 7 7 71HOUCFU AWS Lambda: Function CREATE COMPLETE 
Aat niateway Docke-Attac-1TV81Ax 18 LO VAM AWS ECA “VPCGalewayAtiach CREATE COMPLETE 
CloudstorE SS Policy Docke-Clau-1NMARSCSSKXLI AWS CLARE: Policy CREATE COMPLETE 
DockerL ogórcup Dockerswarm-Ig AWS. Logs: Logoroup CREATE COMPLETE 
OynDBPolic ies Docke-LynD-7 £LOO1H&AY 11 AWS IAM: Policy CREATE COMPLETE 
Syne worker ioc es Doc ke- Dyn LACH FENVVZWO&BVE ANS LAA Policy CREATE COMPLETE 
Externall oadBalancer DockerSwa-Extemal- 1HBH*1HC4D900 AWS ElasticLoadBalancing:-Lo... CREATE COMPLETE 


Figure 3-15. CloudFormation stack resources 
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The Events tab shows the events that occur in creating a CloudFormation stack, as shown in Figure 3-16. 


(p CloudFormation ~ 





Filter: Active * 


Stack Name 
= Dorkerwanm 


Overview Outputs 
2017-07-22 

» 15:02:12 UTC-0700 
+ 150211UTC-0700 
* 1502x110 UTC-0700 
* 150207 UTC-0700 


+ 150406 UTC-0700 
1006 UTC-0700 
* 150202 UTC-O700 
* 1501:43 UTC-O700 


» 98:09:43 UTC-O700 
+o 150138UTC-0700 





CREATE_IN_PROGRESS 
CREATE_IN_PROGRESS 
CREATE_COMPLETE 
CREATE_COMPLETE 


CREATE_IN_PROGRESS 
CREATE_IN_PROGRESS 
CREATE_COMPLETE 
CREATE COMPLETE 


CREATE IN PROGRESS 
CREATE !M PROGRESS 


Stacks 
Actions Design template 
Created Time Status 
2017-07-22 14:52:34 UTC-0700 CREATE IN PROGRESS 
Resources Events Template Parameters Tags 
b 
Status Type 


AWS AutoSealing: AuioSe alingGroup 
AWS: AutoSe aling-:AutoSc alingGroup 
AWS: LAM InstancePrefile 

ANS AutoScaling::LaunchConfiguration 


AWS AuloScaling: LaunthContiguration 
AWS AutoscalingLaunchcConfiguration 
AWS VAM Instance Profile 

AWS IAM Policy 


ANS LAM; -Policy 
ANS AM; Policy 


Figure 3-16. CloudFormation stack events 


Description 


Stack Policy Change Sets 


Logical ID 
ManagerAsg 
Manager&sg 
WorkerlnstanceProfile 


ManagerLaunchConfig17060ce 


awsz 


ManagerLaunchConfig 17060ce 


aus 


Manager aunchConfig 17060ce 


awsz 
ProxyInstanceProfile 
EynDBVvorkerPoli les 
CynDBVorkerPolk ies 
OynDBVeorkerPolic ies 


Docker CE for AWS 17.05. 0-ce (17.05. 0-ce-aws2) 


When the stack creation completes, the status says CREATE COMPLETE, as shown in Figure 3-17. 








@ CloudFormatlon ~ Stacks 
Actions v Design template 
Filter: Active + 
Stack Name Created Time Status 
# DeckerSwanm 2017-07-22 14:55:31 UTC-0700 CREATE COMPLETE 
Overview Outputs Resources Events Template Parameters Tags Stack Policy 
2017-07-22 Status Type 


»od50833UTC-O700 
t 150830 UTC-0700 
t 15:08:29 UTC-0700 
t 15:0829 UTC-0700 
* 150825 UTC-0700 
to 1508253 UTC-0700 


+ 150802 UTC-0700 


Eo 15001 UTC-O700 


» 150801 UTC-0700 


CREATE_COMPLETE 
CREATE_COMPLETE 
CREATE_IN_PROGRESS 
CREATE_IN_PROGRESS 
CREATE_COMPLETE 
CREATE_IN_PROGRESS 


CREATE_IN_PROGRESS 
CREATE_IN_PROGRESS 


CREATE_IN_PROGRESS 


AWS XCbudFormaltion: Sark 

AWS AutoScaling-:LdecycleHock 
AS: AutoScaling-;LiecycleHock 
AWS AutoScaling-LdecycleHock 
ANS:  AutoScaling-AutoSc alingGraup 
AWS--AutoScaling--AutoSc alingSroup 


AWS. AuloScaling Autasc alingGroup 
ANS AUTO Scaling: Autasse alingtroup: 


AWS AuloScaling. AuioScalingSroup 


Figure 3-17. Stack status is CREATE_COMPLETE 
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Description 


Change Sets 


Logical ID 

DockerSwarm 
SwarrmWeorkerporadeHook 
SwannivorkerLipgradeHook 
&warmWorkerLIparadeHook 
NodeAsq 

NodeAsq 


NodeAsg 
NndeAsg 


Nodensg 


Docker CE for AWS 17.06. 0-ce (17.05 0-ce-aws2) 


cou 
Showing 1 Sack 
BEB 
Status reason F 
Resource creation Initiated 
Resource creation initiated 
Resource creation Initiated 
cu 
Showing 1 sack 
BEB 


Status reason 


Resource creation Initiated 


Received SUCCESS signal with Liniqueld i-Daacés1e 
ac5Tf2bd3 

Received SUCCESS signal wilh Uniqueld i-Gadied5a 
24870c3D6 

Received SUCCESS signal with Liniqueld i027 111427 
$042125ba 

Recelved SUCCESS signal wilh Uniqueld i-D4f5c727 

encod de 
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All the required resources—including auto-scaling groups, EC2 Internet Gateway, EC2 security groups, 
Elastic Load Balancer, IAM policy, Log Group, and VPC Gateway—are created, as shown in Figure 3-18. 


(mp CloudFermation ~ Stacks 





Create Stack | ~ ACIS Design template coo 
Filter: Active * showing 1 stack 
Stack Name Created Time Status Description 
X  DockerSwarm 2017-07-22 14:55:31 UTC-0700 CREATE COMPLETE Docker CE for AWS 17.06. 0-ce (17.06. D-ce-awsz) X 
Overview Outputs Resources Events Template Parameters Tags StackPolicy Change Sets mm 
Logical ID Physical ID Type status Status Reason . 
2017/07/22] $LATES T]a^sto58372 7 Jebbb1578bd32 18a 
AZinfo seh nem 5 PPS Custom: AZInfo CREATE COMPLETE 
AziInfoFuncticn DockersSwarm-AZIntoFunction«T8CF £771HCKZFÜ ANS Lambda Fune tiun CREATE COMPLETE 
Aitachialeway Doache-Aultac-1'VS 1s TEU Ta AWS ECS VPC GaiewayAtacn CHEATE COMPLETE 
ChautstorEBS Policy Docke-Clau-TRNNARSCSESRKXLI AWS LAM Policy CREATE COMPLETE 
DockerL ogGroup Dockerswarm-ig ANS: Logs: Log Group CREATE_COMPLETE 
DyniDePolicies Docke-DynD-7 ZLOO 1H9AY1I ANS UAM: Policy CREATE_COMPLETE 
DynDBwWorkerPolicies Docke-DynD-T1ACaTBIMVWOSVB AWS. LAM; Policy CREATE COMPLETE 
bs Exteérnall cadBalanc er DackerSwa-Exlemal- 1HBH31HC4DsCO AWTS. Elastic LoadBalancimg. Lo... CREATE COMPLETE 
ExtemalLoadBalanc ers sg-Obgapera AWS ECZ:SecurityGroug CREATE COMPLETE - 


Figure 3-18. Resources are created 


The Outputs tab lists the Default DNS target, the zone availability comment about the number of 
availability zones, and the manager nodes, as shown in Figure 3-19. 


(D CloudFermation ~ Stacks 





Create Stack | + [Bits ie Design template css 
Filter: Active * Showing 1 stack 
Stack Name Created Time Status Description 
€ DockerSwarm 2017-07-22 14:59:31 UTC-0700 CREATE COMPLETE Docker CE for AWS 17.06.0-ce (17.06. 0-ce-aws2) z 
Overview Outputs Resources Events Template Parameters Tags StackPolicy Change Sets mm 
Key Value Description Export Name 


Dacker&wa-Extemal-THBHS$HHC4D3C 0-485813 


D'efaultDN S Tange 
D 9 68 us-east- 1. elb.amazonaws com 


Use this name to update your DNS records 
This region has at least 3 Availability Zones (A7) 

ZoneAvailability Comment This is ideal to ensure a fully functional Swanm in = Avaltabilty Zones Comment 
tase you lose an AZ 


hips fus-east«1.console aws amazon comer Hw 
zihiame'?region-ws-eas- T&tnstances tag aws.aul 


Managers You can see the manager nodes associated wi. 
Ex oscaling: geoupNamezDockerswanm-ManagerAs j ; 
g-T1LGETSSYZNIC A sort"desc.ánsName 
VEPCID vpc-055d3f7c Lise this as the WPC for configuring Private Ho.. 
ELBDWSZonelD Z355XDOTROTXTK Use this zone ID to update your ONS records 


Figure 3-19. Outputs 
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To list the EC2 instances for the Swarm managers, click on the link in Managers, as shown in Figure 3-20. 


(D CleudFormation ^ Stacks 


Filter: Active * Showing 1 stack 
Stack Name Created Time Status Description 
* DockerSwarm 2017-07-22 14.5931 UTC-0700 CREATE COMPLETE Docker CE for AWS 17.06 0-ce (17.06. O-ce-aws2) x 
Overview Outputs Resources Events Template Parameters Tags StackPolicy Change Sets SFE 
Key Value Description Export Name 


DackerSwa-Extemal-THBH81HC4D9CO-435913 


DefaultDiNS Tange 
9 GB. us-east- 1. elb amazonaws com 


Use this name to update your ONS records 


This region has at least 3 Availability Zones (AZ) 
ZoneAvailatility Comment This is ideal bo ensure a fully functional Swarm in = Avaltabity Jones Comment 
case you lose an AF 





2maoame region us-east- insan es Tag. aws aut 








Managers You can see the manager nodes associated wi 
TILGE TSSY CHIC SA SOT peres upet 1 contcledwis.amazen.cem/ecz/w.iheme region us-east 
Titiristanic esta mei sutosc alin group hames DockerSsarme Banagerlw- 11 BETSSY 28A C A sent = descdeislame 
VPCID vpc-D55d3f7c " z 
ELBDHSZonelD Z355XDOTRQTXTK Use this Zone ID to update vour DNS records 


Figure 3-20. The Managers link 


The three manager instances are all in different availability zones. The public/private IP addresses and 
the public DNS name for each EC2 instance may be obtained from the EC2 console, as shown in Figure 3-21. 


Launch Instance Connect Actions v 
CL o 99 


Q, aws:autoscaling:groupName : DockerSwarm-ManagerAsg-1L8ETS5Y2MC5A Add filter Q il < 1to30f3 > > 
Name + Instance ID ~ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 

f  DockerSwarm-Manager i-0339ef772ffbf2a18 Q micro us-east-1c Q running © 2i2checks.. None `Yo ec2-54-89-68-201 com 
DockerSwarm-Manager i-0029a79d96d0fada2 2 micro us-east-1b @ running © 2/2 checks .. None Ne — 06c2-34-226-138-197 co 
DockerSwarm-Manager — i-03f871b372a1151ad micro us-east-1a @ running © 2/l2checks... None Ye 0c2-34-200-226-246.co 

> 

Instance: [ 1-0339¢t772ftbt2a18 (DockerSwarm-Manager) Public DNS: ec2-54-89-68-201.compute-1.amazonaws.com mmp m - 


Description Status Checks Monitoring Tags 


Instance ID — i-0339cf772f15/2318 Public DNS (IPv4) — ec2-54-89-68-201.compute- 
1.amazonaws.com 
Instance state running b IPv4 Public IP 5489.68 201 
Instance type — 2.micro IPv6 IPs 
Elastic IPs Private ONS — ip-172-31-33-35.ec2. intemal 
Availability zone  us-east-ic Private IPs 172.31.33.35 
Security groups — DockerSwarm-ManagerVpc SG- Secondary private IPs 


1UAGHYRA351UQ , DockerSwarm- 
SwarmWideSG-OWDRBSO4F 087. view 
inbound rules 


Figure 3-21. Manager instances on EC2 
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The AMI used for the EC2 instances may be found using the AMI ID, as shown in Figure 3-22. A Moby 
Linux AMI is used for this Swarm, but the AMI could be different for different users and in different AWS 
regions. 


Launch Instance Connect Actions Y 


o Ff 9 


Q, aws:autoscaling:groupName : DockerSwarm-ManagerAsg-1LSETSSY2MC5A Add filter Q K < 1to30t3 > > 
Name ~ Instance ID ~ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 

@ DockerSwarm-Manager  i-0339cf772ffbf2a18 Q.micro us-east-1c @ running © 2/2checks... None Me  ec2-54-89-68-201.com... 
DockerSwarm-Manager — :-0029a79d96d0fada2 — Q.micro us-east-1b @ running © 2/2checks .. None Me  09c2-34-226-138-197 co 
DockerSwarm-Manager i-OafS71b372a1151ad t2.micro us-east-1a @ running © 2/2checks... None Ne  ec2-34-200-226-246.co 

instance ID 1-0339cf772ffbf2a18 Public DNS (IPv4) — ec2-54-89-68-201 compute- "a 
1.amazonaws com 
Instance state — running IPv4 Public IP — 54 89.68 201 
Instance type  Q.mi«ro IPV6 IPs - 
Elastic IPs Private DNS — ip-172-31-33-35.&c2 internal 
Availability zone — us-east-1c Private IPs 172.31.33.35 
Security groups — DockerSwarm-ManagerVpeSG- Secondary private IPs 
1UAGHYRA351UQ , DockerSwarm- 
SwarmnWideSG-QWDRB6OM4F087 . view 
inbound rules 
Scheduled events No scheduled events VPCID  vpc-055d3f7c 
AMIID Moby Linux 17.06 0-ce-aws2 stable (ami- Subnet!D subnet-d1326099 
È a25c51b4) 
Platform Network interfaces — ethO Y 


Figure 3-22. Moby Linux AMI 


You can list all the EC2 instances by setting Instance State to Running. The Docker Swarm manager 
nodes (three) and worker nodes (five) are listed, as shown in Figure 3-23. The manager and worker nodes 
are in three different availability zones. 


Launch Instance Connect Actions Y 


o 9? 9 


Q, Instance State : Running Add filter @ K < 1toBof8 > > 
Name ~ Instance ID ~ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 

8  DockerSwarm-worker i-Dadfe4522497dc3b6 2. micro us-east-1a @ running © 22checks... None Ww ec 2-107-23-82-165.co.... 
DockerSwarm-worker i-O4f6c727e2c94384e — Q. micro us-east-1b @ running © 2/2 checks.. None YW  ec2-54-173-99-220.co 
DockerSwarm-worker i-OaacS8teacS7f2bd3 — Q micro us-east-1c @ running © 2/2checks... None We ec 2-54-144-60-220.co... 
DockerSwarm-Manager  i-0339cf772ffbi2a1s Q micro us-east-1c @ running © 2/2checks... None Me —0c2-54-89-68-201.com 
DockerSwarm-worker i-027111d2804a125ba — Q.micro us-east-1c Q running © 2/2checks... None We  06c2-52-23-203-212.co 
DockerSwarm-Manager — :-0029a79d96d0fada2 t2.micro us-east-1b @ running © 2/2 checks None Ne — 9c2-34-226-138-197 co 
DockerSwarm-worker i-Oibb468a3a5babS4c — (2. micro us-east-1a @ running © 2/2checks... None Yo ec 2-34-205-53-11.com.. 
DockerSwarm-Manager — :-0af871b372a1151ad — OQ micro us-east-1a @ running Q 2/2checks... None Ng ec 2-34-200-226-246.co 





Instance: J i-034fe45a2497dc3b6 (DockerSwarm-worker) ^ Public DNS: €¢2-107-23-82-165.compute-1.amazonaws.com mmm:- 
Description Status Checks Monitoring Tags 
Instance ID — i-0a4fe45a2497dc3b6 Public DNS (IPv4) — ec2-107-23-82-165 compute- 
1 amazonaws com 
Instance state — running N IPv4 Public IP 107.23.82.165 
Instance type micro IPv6 IPs 


Figure 3-23. Swarm managers and workers in three different availability zones 
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Select Load Balancers in the EC2 dashboard and the provisioned Elastic Load Balancer is listed, as 
shown in Figure 3-24. Click on the Instances tab to list the instances. All instances should have a status set to 
InService, as shown in Figure 3-24. 


Create Load Balancer Actions v 
M create Load Balancer o 


e 0 
Filter: © Search X 1to120f1 | 
@ Name = DNS name - State - VPC ID = Availability Zones = Type 
b @ OcckerSwa-Extemal-1HBHS... DockerSwa-External-THBH3... vpe-D55d 3c us-easi-1a,. us-east-1b... classic 
L] 
Load balancer: | DockerSwa-External-1HBH91HC4D9C 0 SRE. 
Description Instances Heath Check Listeners Monitoring Tags 
Connection Draining: Disabled (Edit) 
Edit Instances 
Instance ID Name Availability Zone Status Actions 
i-0339cf7721fb(2a18 DockerSwarm-Manager us-east-1c InSerace C1 Remove from Load Balancer 
-Uaf 1b372a1151ad Dockerswarm-Manadger us-east-1a Insenace È Remove from Load Balancer 
i-DAfec ! 2782094 3848 LDockerSwarm-worker us5-east-1h InService | |) Remove from Load Balancer 
i-Oibb4bSata5babsde Dockerswarm-worker us-east-1a InSereice (| Remove from Load Balancer 
i-Dadfgodba2487dc 3b6 DockerSwarm-worker us-east-Ta Inservice (I Remove from Load Balancer 
i-Q029a79d96d0Ofada? DockerSwarm-Manager us-east-1h InSeráce (1 Remove from Load Balancer 
i-Qa7111d2804a125ba DockerSwarm-worker us-east-1c InSereace (i Remove from Load Balancer T 


Figure 3-24. Elastic Load Balancer 


Select Launch Configurations from the EC2 dashboard. The two launch configurations—one for the 
managers and one for the worker nodes—will be listed, as shown in Figure 3-25. 


Create launch configuration Actions y 
o ¢ @ 


4 


Filter: Q Filter launch conf 1 to 2 of 2 Launch Configurations 
Name a AMIID Instance Type ~ Spot Price Creation Time 
DockerSwarm-Nodel aunchConfig 17060ceaws2-RYA1BSVJKDF 3 ami-a25c51b4 Q micro July 22. 2017 30529 PM UTC-7 
k DockerSwarm-ManagerLaunchConfig17060ceaws2-1VOMPPGK1XMI ami-a25c5 1b4 Q micro July 22. 2017 3:02:06 PM UTC-7 


Figure 3-25. Launch configurations 


Select Auto Scaling Groups in the EC2 dashboard. The two auto-scaling groups—one for the managers 
and one for the worker nodes—will be listed, as shown in Figure 3-26. 


Create Auto Scaling group Actions v 
IB Create Auto Scaling group | o 90 


Filter: Q Filler Auto Scaling groups x 1 to 2 of 2 Auto Scaling Groups 
Name a Launch Configuration ~ Instances ~ Desired - Min ~ Max ~ Availability Zones ~ Default Cooldow 
DockerSwarm-NodeAsg-1PWVVETKVWXMJ DockerSwarm-NodeLau § 5 0 1,000 us-east-1a, us-east-1b, us-e 300 
M DockerSwarm-ManagerAsg-1LOETSS5Y2MC5A DockerSwarm-Manager 3 3 0 6 uS-@ast-1a, us-east-1b, us-e 300 


Figure 3-26. Auto-scaling groups 
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Connecting with the Swarm Manager 


Next, connect to a Swarm manager node from the local machine on which the key pair docker. pem is copied. 
Using the public IP address of a manager EC2 instance, SSH login into the instance with user as “docker” 


ssh -i "docker.pem" docker@54.89.68.201 

The command prompt for the manager node is displayed. 
[root@localhost ~]# ssh -i "docker.pem" docker@54.89.68.201 
Welcome to Docker! 


The Docker version of the Swarm node may be listed using docker --version. The version will be 17.06 
or greater. Swarm mode is supported on Docker 1.12 or greater. 


~ $ docker --version 
Docker version 17.06.0-ce, build 02c1d87 


Using the Swarm 


List the Swarm nodes with docker node ls and the three manager nodes and five worker nodes will be 
listed. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 

25511m8729rns82bmloaxs6usl ip-172-31-8-37.ec2.internal Ready Active 
ikyskl4ysocymoe4pbrj3qnh3 ip-172-31-4-154.ec2.internal Ready Active Reachable 
p2ky6meej8tnph5wyuw59xtmr ip-172-31-21-30.ec2.internal Ready Active Leader 
r56kkltfgc4zzzfbslinrun2d1 ip-172-31-24-185.ec2.internal Ready Active 
sogez5qplcihk8y2y58uj9md4 ip-172-31-1-33.ec2.internal Ready Active 
xbdeo8qp9jhi398h478wl2zrv * ip-172-31-33-35.ec2.internal Ready Active Reachable 
ykk4odpjps6t6eqc9mriqvo4a ip-172-31-47-162.ec2.internal Ready Active 
zrlrmijyj5vklxl3ag7gayb3w ip-172-31-39-210.ec2.internal Ready Active 


The leader node and two other manager nodes indicated by Manager Status of Leader and Reachable 
are listed. The worker nodes are all available, as indicated by Active in the Availability column. 
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Docker services are introduced in the next chapter, but you can run the following docker service 
create command to create an example Docker service for a MySQL database. 


docker service create \ 
--env MYSQL ROOT PASSWORD=' mysql’ \ 
--replicas 1 \ 
--name mysql \ 
--update-delay 10s \ 
--update-parallelism 1 \ 

mysql 


A service gets created: 


~ $ docker service create ^ 
--env MYSQL ROOT PASSWORD= 'mysql'N 
--replicas 1 \ 
--name mysql \ 
--update-delay 10s \ 
--update-parallelism 1 \ 
mysql 
12hg71a3vy793quvi4uems5gk 


V N OV VV ON 


List the service with the docker service ls command, which is also discussed in the next chapter, and 
the service ID, mode, replicas, and image are listed. 


~S docker service ls 
ID NAME MODE REPLICAS IMAGE 
n2tomumtl9sbniysql replicated 1/1 mysql:latest 
Scale the service to three replicas with the docker service scale command. The three replicas 
are scheduled—one on the leader manager node and two on the worker nodes. The docker service ps 
command to list service replicas is also discussed in more detail in the next chapter. 
~ S docker service scale mysql-3 
mysql scaled to 3 
~ S docker service ps mysql 
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 
slqtuf9l4hxo mysqi.1 mysql:latest ip-172-31-35-3.us-east-2.compute. internal 
Running Running about a minute ago 
exqsthrgszzc mysql.2 mysql:latest ip-172-31-27-83.us-east-2.compute. internal 
Running Preparing 8 seconds ago 


vtuhsl6mya85 mysql.3 mysql:1atest ip-172-31-29-199.us-east-2.compute.internal Running 
Preparing 8 seconds ago 
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Deleting a Swarm 


To delete a Swarm, choose Actions »* Delete Stack from the CloudFormation console, as shown in Figure 3-27. 


(D CloudFormation ~ Stacks 





- Actions + Design template e 2 








Filter: Active v Create Change Set For Current Stack Showing 1 stack 
pdate Stac 
Stack Name Update Stack Status Description 
* DockerSwarm € 0 CREATE COMPLETE Docker CE for AWS 17.06 .0-ce (17.06.0-ce-aws2) 
View Edit template in Designer 
Overview Outputs Resources Events Template Parameters Tags StackPolicy Change Sets m oe Bh | 


Stack name: DockerSwarm 
Stack!D:  arm:aws cloudformation:us-east-1.672593526585.stack/DockerSwarm/09ce3260-6129- 11e7-9c71-500c28527a35 
Status: CREATE CCMPLETE 
Status reason: 
IAM Role: 


Description Docker CE for AWS 17 06.0.ce (17.06.0.ce-aws2) 
Figure 3-27. Choosing Actions > Delete Stack 


In the Delete Stack confirmation dialog, click on Yes, Delete, as shown in Figure 3-28. 










Delete Stack x 


Are you sure you want to delete this stack? 
DockerSwarm 


Deleting a stack deletes all stack resources. 


Cancel Yes, Delete 


+ 


Delete Stack 


Figure 3-28. Delete stack confirmation dialog 
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The stack’s status becomes DELETE IN PROGRESS, as shown in Figure 3-29. 


(p CloudFormation ~ Stacks 





Actions v Design template C uu 
Filter: Active v Showing 1 stack 
Stack Name Created Time Status Description 
* DockerSwarm 2017-07-22 14:59:31 UTC-0700 DELETE_IN PROGRESS Docker CE for AWS 17.06.0-ce (17.06.0-ce-aws2) 


Figure 3-29. Delete in progress 


As each of the stack's resources is deleted, its status becomes DELETE COMPLETE, as shown for some of 
the resources on the Events tab in Figure 3-30. 


(D CloudFormation v Stacks 








Design template C o 
Showing 1 stack 
Stack Name Created Time Status Description 
+ DockerSwarm 2017-07-22 14:59:31 UTC-0700 DELETE IN PROGRESS Docker CE for AWS 17 06 0-ce (17 06 0-ce-aws2) 
Overview Outputs Resources Events Template Parameters Tags StackFolicy Change Sets aag 
2017-07-22 Status Type Logical ID Status reason 
» 15:38:24 UTC-0700 DELETE IN PROGRESS AWS..EC2: Subnet PubSubnetAz2 
> 15:38:24 UTC-0700 DELETE_IN PROGRESS AWS..EC2: VPCGaleway Attac hment AttachGateway 
>» 15:38:24 UTC-0700 DELETE IN PROGRESS AWS::EC2:-:Subnet PubSubnetAz1 
b 15:38:24 UTC-0700 DELETE IN PROGRESS AWS-:EC2: Subnet PubSubnetaz3 
b 15:38:22 UTC-0700 DELETE COMPLETE AWS:-ElasticLoadBalancing: LoadBalancer ExternalLoadBalancer 
> 15:3822 UTC-0700 DELETE IN PROGRESS AWS-ElasbicLoadSalancing: LoadBalancer ExternalLoadBalancer 
>» 153820UTC-0700 DELETE COMPLETE AWS-'DynamoDB "Table SwarmDynDB Table 
» 153754UTC-0700 DELETE COMPLETE AWS- IAM: Role ProxyRole 
> 15:37:53 UTC-0700 DELETE_IN_PROGRESS AWS-1AM:: Role ProxyRole 
>» 15:37:53 UTC-0700 DELETE COMPLETE AWS..EC2. Security Group Nodevoc SG 
> 15:37:52 UTC-0700 DELETE IN PROGRESS AWS..EC2. Security Group NodeVpX SG 





Figure 3-30. Events list some of the resources with a status of DELETE COMPLETE 
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When the EC2 instances have been deleted, the EC2 console lists their status as terminated, as shown 


in Figure 3-31. 


Launch Instance Actions v 
4 
Q, Filter by tags and attributes or search by ki 
Name Instance ID 
DockerSwarm-Manager — i-0029379d96d0fada2 


DockerSwarm-worker 
DockerSwarm-worker 
DockerSwarm-Manager 
DockerSwarm-worker 
DockerSwarm-worker 
DockerSwarm-worker 


DockerSwarm-Manager 


-01bb468a3a5bab84c 


-02711182804a125ba 
40339cf772105f2218 

i-04f6c727e2c94384e 
i-0a4fe4522497dc3b6 
i-Oaac88teacS7f2bd3 


i-0af87 1b372a1151ad 


Instance Type ~ 


12.micro 
(2. micro 
(2. micro 
t2 micro 
12 micro 
t2.micro 
t2. micro 


t2 micro 


Availability Zone ~ 


us-east- 1b 
us-east-1a 
us-east-1c 
us-east-1c 
us-east-1b 
us-east-1a 
us-east-1c 


us-east-1a 


Figure 3-31. EC2 instances with status set to terminated 


Summary 


Instance State ~ 


@ terminated 
Q terminated 
Q terminated 
Q terminated 
Q terminated 
Q terminated 
Q terminated 


Q terminated 


Status Checks - 


9 


Alarm Status 


None 


&/ os s/os wow wow 


e 


1108 of 8 


o o0 


Public DNS (IPv4) 


This chapter discussed creating a multi-zone Docker Swarm provisioned by a CloudFormation template 
using the Docker for AWS service. You learned how to connect to the Swarm manager to run docker 
service commands. The next chapter introduces Docker services. 
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Docker Services 





A Docker container contains all the binaries and dependencies required to run an application. A user only 
needs to run a Docker container to start and access an application. The CoreOS Linux operating system has 
Docker installed and the Docker commands may be run without even installing Docker. 


The Problem 


A Docker container, by default, is started only on a single node. However, for production environments, 
where uptime and redundancy matters, you need to run your applications on multiple hosts. 

When a Docker container is started using the docker run command, the container starts only on 
a single host, as illustrated in Figure 4-1. Software is usually not designed to run on a single host only. A 
MySQL database in a production environment, for example, may need to run across a cluster of hosts for 
redundancy and high availability. Applications that are designed for a single host should be able to scale up 
to multiple hosts as needed. But distributed Docker applications cannot run on a single Docker Engine. 


Docker 
Engine 


docker run -d -p 
8080 tututm/ Docker 


hello-world Conainer 





Figure 4-1. Docker container on a single host 


The Solution 


Docker Swarm mode enables a Docker application to run across a distributed cluster of Docker Engines 
connected by an overlay network, as illustrated in Figure 4-2. A Docker service may be created with a specific 
number of replicas, with each replica potentially running on a different host in a cluster. A Swarm consists of 
one or more manager nodes with a single leader for Swarm management and orchestration. Worker nodes 
run the actual service tasks with the manager nodes being worker nodes by default. A Docker service may 

be started only from the leader node. Service replicas scheduled on the worker nodes, as a result, run a 
distributed application. Distributed applications provide several benefits, such as fault tolerance, failover, 
increased capacity, and load balancing, to list a few. 
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r 1 Docker 
I I 


helloworld.1 Container 


Swarm 
Manager 
r 1 Docker 
[| I 


helloworld.2 Container 


3 tutum/hello-world 
service replicas 


Docker 


r 
I Container 


helloworld.3 





Figure 4-2. Docker service tasks and containers spread across the nodes 


This chapter covers the following topics: 
e Setting the environment 
e The Docker service commands 
e Types of services 
e Creating a service 
e Listing the tasks of a service 
e Invoking a Hello World service task on the command line 
e Getting detailed information about a service 
e Invoking the Hello World service in a browser 
e Creating a service for a MySQL database 
e Scaling a service 
e Listing service tasks 
e Accessing a MySQL database in a Docker container 
e Updating a service 
e Updating the replicas 
e Updating the Docker image tag 


e Updating the placement constraints 
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e Updating environment variables 
e Updating the Docker image 

e Updating the container labels 

e Updating resources settings 


e Removing a service 


Setting the Environment 


Create a Docker Swarm consisting of one manager and two worker nodes using the procedure discussed in 
Chapter 3. First, start three CoreOS instances—one for a Swarm manager and two for the Swarm workers. 
Obtain the public IP address of the Swarm manager, as shown in the EC2 console in Figure 4-3. 





Wn 2E M Connect ^ Actions ¥ 
O esses rr o 9 @ 
Cl Instance State : Running Add filter o 1tod3oft3 
Name - Instance ID * Instance Type ~ Availability Zone ~ Instance State ~ — Status Checks ~ Alarm Status Public ONS [IPv4] 
Deckerswarm-worker -ÜSfe*aab/Üeadfc3b — 12 micro us-east-1c @ running @ 2/2 checks None P ec2-52-31-39-27b com 
DockarSwarm-worker :n4d06a4d265a89 12 micro us-east-1a @ running @ 2/2 checks None Ya ee2-52-64-70-201 com 
E  DockerSwarm-Manager — i-0436f9d57d0d850d9 — (2. micro us-east-1a Q running © 2/2 checks None Me 002-24-200-225-33 co... 
k 
Instance: | i-0436f9d57d0d950d9 (DockerSwarm-Manager) Public ONS: ec2-24-200-225-38.compute-1.amazonaws.com BEE - 
Description Status Checks Monitoring Tags 
Instance ID — i-0436f9d57d0d950d9 Public ONS (IPv4) — ec2-34-200-225-38.compute- 
1.amazonaws.com 
Instance state — running IPvá Public IP — 34 200.225 38 
Instance type — 2 micro [Pi IP 
Elastic IPs Private ONS — ip-172-31-13-155.ec2 intemal 
Availability zone ^ us-east-la Private IPs — 1/2.31.13.155 
Security groups — DackerSwarm-ManagerVpcSG- Secondary private IPs 
1TB4UTOFS6TSA, DockerSwarm- 
SwarmWideSG-1HIEMLHISIDAH . view 
inbound rules - 


Figure 4-3. EC2 instances for Swarm 


SSH login to the Swarm manager instance with user as “docker’ 


[root@localhost ~]# ssh -i  "docker.pem" docker@34.200.225.39 
Welcome to Docker! 


Three nodes should get listed in the Swarm with the docker node 1s command—one manager node 
and two worker nodes. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
ilru4foi280w2tlsrg9hglwsj ^ ip-172-31-10-132.ec2.internal Ready Active 
w5toi86ipblpcq390625wyq2e | ip-172-31-37-135.ec2.internal Ready Active 
zkxle7kafwcmtisd93kh5cy5e * ip-172-31-13-155.ec2.internal Ready Active Leader 


57 


CHAPTER 4 ~ DOCKER SERVICES 


A worker node may be promoted to a manager node using the docker node promote <node ip> 
command. 


~ $ docker node promote ilru4f0i280w2tlsrg9hglwsj 
Node ilru4f0i280w2tlsrg9hglwsj promoted to a manager in the swarm. 


If you list the nodes again, two manager nodes should be listed. A manager node is identified by a value 
in the Manager Status column. One node has a Manager Status of Reachable and the other says Leader. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
ilru4foi280w2tlsrg9hglwsj ^ ip-172-31-10-132.ec2.internal Ready Active Reachable 
w5to186ipblpcq390625wyq2e ip-172-31-37-135.ec2.internal Ready Active 
zkxle7kafwcmtisd93kh5cy5e * ip-172-31-13-155.ec2.internal Ready Active Leader 


The manager node that is the Leader performs all the swarm management and orchestration. The 
manager node that is Reachable participates in the raft consensus quorum and is eligible for election as the 
new leader if the current leader node becomes unavailable. 

Having multiple manager nodes adds fault tolerance to the Swarm, but one or two Swarm managers 
provide the same fault tolerance. If required, one or more of the worker nodes could also be promoted to a 
manager node to increase fault tolerance. 

For connectivity to the Swarm instances, modify the inbound rules of the security groups associated 
with the Swarm manager and worker instances to allow all traffic. The inbound rules for the security group 
associated with a Swarm node are shown in Figure 4-4. 


Create Secu Grou Actions Y 
gl Create Securty Group | oto 





Q, search : sg-6c39101d Add filter &} 1t02 of2 
Name ~ Group ID ^ Group Name ~ VPCID ~ Description 
id $g-6c39101d DockerSwarm-NodeVpcSG-.. vpc-b9bed3cO Node SecurityGroup 
$s3-dc250cad DockerSwarm-ManagerVpc vpc-bSbed3cO Manager SecurityGroup 
Security Group: sg-6c39101d mmrmt 
Description Inbound Outbound Tags 
lo 
Edit 
Type ii Protocol |i Port Range |i Source ‘i 
All traffic All Al 0.0. 0.0/0 


All traffic All All 0 


Figure 4-4. Setting inbound rules on a security group to allow all traffic 
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The outbound rules for the security group associated with the Swarm manager are shown in Figure 4-5. 


Create Security Group Actions Y 
< 


Q, search : sg-de250cad 


Name 


Group ID 


$g-dc250cad 


Security Group: sg-dc250cad 


Description 


Edit 


Type (i 


N All traffic 


Inbound 


Outbound 


o 9 9 
o 1to 1 0f 1 
^ Group Name VPC ID Description 
DockerSwarm-ManagerVpc vpc-b9bed3cO Manager SecurityGroup 
m mr 

Tags 
Protocol ʻi Port Range ʻi Destination (i 
All All 0.0.0.0/0 


Figure 4-5. Setting outbound rules on a security group to allow all traffic 


The docker service Commands 


The docker service commands are used to manage Docker services. The docker service command 
provides the sub-commands listed in Table 4-1. 


Table 4-1. The docker service Sub-Commands 


Command 


docker 
docker 
docker 
docker 
docker 
docker 
docker 


docker 


service 
service 
service 
service 
service 
service 
service 


service 


create 
inspect 
logs 

ls 

ps 

rm 
scale 


update 


Description 


Creates a new service. 

Displays detailed information on one or more services. 

Fetches the logs of a service. The command was added in Docker 17.0.6. 
Lists services. 

Lists the tasks of one or more services. 

Removes one or more services. 

Scales one or multiple replicated services. 


Updates a service. 


To run docker service commands, the following requirements must be met. 


the Leader 


The Docker Swarm mode must be enabled 


The docker service commands must be run from the Swarm manager node that is 


The docker service commands are available only in Swarm mode and cannot be run outside the 
Swarm mode. 

The docker service commands cannot be run from a worker node. Worker nodes cannot be used to 
view or modify Swarm cluster state. 
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Types of Services 


Docker Swarm mode supports two types of services, also called service modes—replicated services and 
global services. Global services run one task only on every node in a Docker Swarm. Replicated services run 
as a configured number of tasks, which are also referred to as replicas, the default being one. The number of 
replicas may be specified when a new service is created and may be updated later. The default service type is 
a replicated service. A global service requires the - -mode option to be set to global. Only replicated services 
may be scaled; global services cannot be scaled. 

We start off by creating a replicated service. Later in the chapter, we also discuss creating a global 
service. 


Creating a Service 


The command syntax to create a Docker service is as follows. 
docker service create [OPTIONS] IMAGE [COMMAND] [ARG...] 


Some of the supported options are listed in Table 4-2. 


Table 4-2. Supported Options for Creating a Service 


Option 


--constraint 


--container-label 


Description 


Placement constraints. 


Container labels. 


--env, -e Sets environment variables. 

--env-file Reads in a file of environment variables. Option not added until Docker 
1.13. 

--host Sets one or more custom host-to-IP mappings. Option not added until 
Docker 1.13. Format is host: ip. 

--hostname Container hostname. Option not added until Docker 1.13. 

--label, -1 Service labels. 

- -limit-cpu Limits CPUs. Default value is 0.000. 


- -limit-memory 


Limits memory. Default value is 0. 


- -log-driver Logging driver for service. 

--log-opt Logging driver options. 

--mode Service mode. Value may be replicated or global. Default is replicated. 
--mount Attaches a filesystem mount to the service. 

--name Service name. 

--network Network attachments. By default, the "ingress" overlay network is used. 
--publish, -p Publishes a port as a node port. 

--read-only Mounts the container’s root filesystem as read only. Option not added 
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until Docker 17.03. 
Default is false. 


(continued) 


Table 4-2. (continued) 
Option 

--replicas 
--reserve-cpu 
--reserve-memory 
--restart-condition 
--restart-delay 
--restart-max-attempts 
--tty, -t 


- -update-delay 


--update-failure-action 
--update-monitor 
--update-parallelism 


--user, -u 


--workdir, -w 
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Description 


Number of tasks. 

Reserves CPUs. Default is 0.000. 

Reserves memory. Default is 0. 

Restarts when condition is met. Value may be none, on-failure, or any. 
Delays between restart attempts (ns|us|ms|s|m|h). 

Maximum number of restarts before giving up. 


Whether to allocate a pseudo-TTY. Option not added until Docker 1.13. 
Default is false. 


Delays between updates (ns|us|ms|s|m|h). Default is Os. 


Action on update failure. Value may be pause or continue. Default value is 
pause. 


Duration after each task update to monitor for failure (ns|us|ms|s|m]|h). 
Default is Os. 


Maximum number of tasks updated simultaneously. A value of 0 to 
updates all at once. Default value is 1. 


Username or UID in format: «name |uid»[ : «group|gid» ]. 


Working directory inside the container. 


As an example, create a service called hello-world with Docker image tutum/hello-world consisting 
of two replicas. Expose the service on port 8080 on the host. The docker service create command outputs 


a service ID if successful. 


~ $ docker service create \ 
>  --name hello-world \ 


> | --publish 8080:80 \ 
>  --replicas 2 \ 
>  tutum/hello-world 


vyxnpstt351124h12niqm7s64 


A service gets created. 


Listing the Tasks of a Service 


You can list the service tasks, also called replicas in the context of a replicated service, with the following 


command. 


docker service ps hello-world 
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The two service tasks are listed. 


~ $ docker service ps hello-world 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

Zjm03bjsqyhp hello-world.1 tutum/hello-world:latest ^ ip-172-31-10-132.ec2.internal 
Running Running 41 seconds ago 


kezidi820l5c hello-world.2  tutum/hello-world:latest ^ ip-172-31-13-155.ec2.internal 
Running Running 41 seconds ago 


The ID column lists the task ID. The task name is in the format servicename.n; hello-world.1 and 
hello-world.2 for the two replicas. The Docker image is also listed. The NODE column lists the private DNS 
of the node on which the task is scheduled. The DESIRED STATE is the state that is desired as defined in the 
service definition. The CURRENT STATE is the actual state of the task. At times, a task could be in a pending 
state because of lack of resource capacity in terms of CPU and memory. 

A service task is a slot for running a Docker container. On each node on which a task is running, a 
Docker container should also be running. Docker containers may be listed with the docker ps command. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

Occdcde64e7d tutum/hello-world:latest "/bin/sh -c 'php-f..." | 2 minutes ago 
Up 2 minutes 80/tcp hello-world.2.kezidi82015ct81u59jpgfhs1 


Invoking a Hello World Service Task on the Command Line 


Invoke the hello-world service using curl at «hostname» :8080. The curl command output is the HTML 
markup for the service. 


~ $ curl ec2-34-200-225-39.compute-1.amazonaws .com: 8080 
«html» 
«head» 
«title»Hello world!«/title» 
«link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700' rel-'stylesheet' 
type-'text/css'» 
«style» 
body 1 
background-color: white; 
text-align: center; 
padding: 50px; 
font-family: "Open Sans","Helvetica Neue",Helvetica,Arial,sans-serif; 


j 


#logo { 
margin-bottom: 40px; 
j 


</style> 


</head> 
<body> 
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«img id-"logo" src-"logo.png" /» 
<hi>Hello world!«/hi» 
«h3»My hostname is 20b121986df6«/h3» 


Getting Detailed Information About a Service 


To get detailed information about the hello-world service, run the docker service inspect command. 


docker service inspect hello-world 


The detailed information includes the container specification, resources, restart policy, placement, 
mode, update config, ports (target port and published port), virtual IPs, and update status. 


~ $ docker service inspect hello-world 


{ 


"ID": "vyxnpstt351124h12niqm7s64", 
"Version": { 
Index": 30 


"CreatedAt": "2017-07-23T19:00:09.98992017Z", 
"UpdatedAt": "2017-07-23T19:00:09.993001487Z", 


"Spec": { 
"Name": "hello-world", 
"Labels": {}, 


"TaskTemplate": { 
"ContainerSpec": { 
"Image": "tutum/hello-world:latestQsha256:0d57def8055178aafb4c7669cbc25e 
c17f0acdab97cc587f30150802da8f84085" , 
"StopGracePeriod": 10000000000, 
"DNSConfig": {} 


Jo 

"Resources": { 
"Limits": {}, 
"Reservations": {} 

) 


"RestartPolicy": { 
"Condition": "any", 
"Delay": 5000000000, 
"MaxAttempts": O 
Jo 
"Placement": { 
"Platforms": [ 
{ 
"Architecture": "amd64", 
"OS": "linux" 


Js 
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"ForceUpdate": 0, 
"Runtime": "container" 


)» 
"Mode": { 
"Replicated": { 
"Replicas": 2 
j 
Js 


"UpdateConfig": { 
"Parallelism": 1, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": O, 
"Order": "stop-first" 

Js 

"RollbackConfig": { 
"Parallelism": 1, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": O, 
"Order": "stop-first" 

Js 

"EndpointSpec": { 

"Mode": "vip", 
"Ports": [ 
{ 
"Protocol": "tcp", 
"TargetPort": 80, 
"PublishedPort": 8080, 
"PublishMode": "ingress" 


j 
] 
j 
J 
"Endpoint": { 
"Spec": { 
"Mode": "vip", 
"Ports": [ 
{ 
"Protocol": "tcp", 
"TargetPort": 80, 
"PublishedPort": 8080, 
"PublishMode": "ingress" 
j 
] 
Js 
"Ports": [ 
{ 


"Protocol": "tcp", 
"TargetPort": 80, 
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"PublishedPort": 8080, 
"PublishMode": "ingress" 
} 
J> 
"VirtualIPs": [ 


"NetworkID": "y3k655bdlp3x102a2bslh4swh", 
"Addr": "10.255.0.5/16" 


Invoking the Hello World Service in a Browser 


The Hello World service may be invoked in a browser using the public DNS of a EC2 instance on which a 
Swarm node is hosted. A service replica does not have to be running on a node to invoke the service from the 
node. You obtain the public DNS of a manager node from the EC2 console, as shown in Figure 4-3. Invoke 
the Hello World service with <Public DNS>:<Published Port> URL. As the Hello World service is exposed 
or published on port 8080, the URL to invoke in a browser becomes <Public DNS>: 8080. The service is 
invoked and the service output is displayed in the browser, as shown in Figure 4-6. 


[) Hello world! x 


Mx C | © ec2-34-200-225-39.compute-1.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is Occdcde64e7d 





Figure 4-6. Invoking a service in a browser 
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Similarly, you can obtain the public DNS of a EC2 instance on which a Swarm worker node is hosted, as 
shown in Figure 4-7. 


Temes Connect Actions + 
E o 90 


C) Instance State : Running Add filter Q Kk 4 1to3of3 > H 
Hame - Instance ID ~ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public ONS (IP) 
DockerSwarm-worker iOStesaabiOeadies5 tmicro us-east-1c Q running © 2/2 checks Nene ‘we 6c2-52-91-38-226 com 

B) — DockerSwarm-worker iüddO6add265e89 . tA micro us-east-1a Q running © 22checks.. Mone Mg —0c2-52-54-70-201 com . 
DockerSwarm-Manager — »0436f9d57d0d350d8 — Q2 miro us-aast-1a @ running © 22checks .. None Ms  8c2-34-200-225-39 co... 

k 

Instance: | i-014d06add265e8918 (DockerSwarm-worker) ^ Public DNS: ec2-52-54-70-201.compute-1.amazonaws.com mp m m 


Figure 4-7. Obtaining the public DNS for a EC2 instance on which a Swarm worker node is hosted 


Invoke the service using the PublicDNS: 8080 URL in a browser, as shown in Figure 4-8. 





D Hello world! 


— C | @ ec2-52-54-70-201.compute-Lamazonaws.com:8080 


tutum 


Hello world! 


My hostname is Occdcde64e7d 





Figure 4-8. Invoking a service in a browser using public DNS for a EC2 instance on which a Swarm worker 
node is hosted 


A manager node is also a worker node by default and service tasks also run on the manager node. 
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Creating a Service for a MySQL Database 


Next, we create a service for a MySQL database. Using the mysql Docker image is different than using the 
tutum/hello-world Docker image in two respects. 


e The mysql Docker image has a mandatory environment variable called MYSQL ROOT 
PASSWORD. 


e Themysql Docker image is based on a Debian Linux and starts the MySQL database 
server in Docker container, while the tutum/hello-world image is based on Alpine 
Linux and starts Apache Server to run PHP applications. 


Run the following docker service create command to create one replica of the MySQL database 
service. Supply a root password with the MYSOL ROOT PASSWORD environment variable. Include some other 
options for the restart condition, the restart max attempts, the update delay, and the update failure action. 
Remove any previously running Docker service called mysql with the docker service rm mysql command. 


~ $ docker service create \ 
--env MYSOL ROOT PASSWORD-'mysql'N 
--replicas 1 \ 
--restart-condition none \ 
--restart-max-attempts 5 \ 
--update-failure-action continue \ 
--name mysql \ 
--update-delay 10s \ 

mysql 


A service gets created for MySQL database and the service ID gets output. 


~ $ docker service create ^ 
--env MYSQL ROOT PASSWORD= 'mysql'N 
--replicas 1 \ 
--restart-condition none \ 
--restart-max-attempts 5 \ 
--update-failure-action continue \ 
--name mysql \ 
--update-delay 10s \ 

mysql 

gz18k1wy8kf3msi1nu5zwlfxmó 


> 
> 
> 
> 
> 
> 
> 
> 


List the services with the docker service ls command; the mysql service should be listed. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
gzl8kiwy8kf3 mysql replicated 1/1 mysql:latest 
vyxnpstt3511 hello-world replicated 2/2 tutum/hello-world:latest *:8080->80/tcp 


List the service tasks/replicas with the docker service ps mysql command. One task is running on 
the manager worker node. 
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~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
mfw76m4rxbhp mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 

Running Running 16 seconds ago 


How service tasks are scheduled, including node selection based on node ranking, is discussed in 
Chapter 8, which covers scheduling. 


Scaling a Service 


Next, we scale the mysql service. Only replicated services can be scaled and the command syntax to scale 
one or more services is as follows. 


docker service scale SERVICE=REPLICAS [SERVICE=REPLICAS... | 

To scale the mysql service to three tasks, run the following command. 
docker service scale mysql=3 

The mysql service gets scaled to three, as indicated by the command output. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


Listing Service Tasks 


The docker service ps command syntax to list service tasks is as follows. 
docker service ps [OPTIONS] SERVICE [SERVICE...] 
The command supports the options listed in Table 4-3. 


Table 4-3. Options for the docker service ps Command 
Option Description 
--filter, -f Filters output based on conditions provided. The following filters are supported: 


id=<task id» 
name=<task name» 
node-«node id or name» 


desired-state-(running | shutdown | accepted) 


--no-resolve Whether to map IDs to names. Default value is false. 

--no-trunc Whether to truncate output. Option not added until Docker 1.13. Default value is 
false. 

--quiet, -q Whether to only display task IDs. Option not added until Docker 1.13. Default value is 
false. 
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As an example, you can list only the service tasks that are running. 
docker service ps -f desired-state-running mysql 
Only the running tasks are listed. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

mfw76m4rxbhp mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 46 seconds ago 

s4flvtode8od mysql.2 mysql:latest ip-172-31-13-155.ec2. internal 
Running Running 8 seconds ago 

j0jd92p5dmd8 mysql .3 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 9 seconds ago 


All tasks are running; therefore, the effect of using the filter is not very apparent. But, in a subsequent 
example, you'll list running service tasks when some tasks are not running. 

Not all worker nodes are utilized for running service tasks if the number of nodes is more than the 
number of tasks, as when the hello-world and mysql services had fewer than three tasks running. A node 
could have more than one service task running if the number of replicas is more than the number of nodes 
in a Swarm. Scaling up to five replicas starts more than one replica on two of the nodes. 


~ $ docker service scale mysql=5 
mysql scaled to 5 
~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

mfw76m4rxbhp mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running about a minute ago 

s4flvtode8od mysql.2 mysql:latest ip-172-31-13-155.ec2. internal 
Running Running 44 seconds ago 

j0jd92p5dmd8 mysql .3 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 45 seconds ago 

vh9qxhm452pt mysql.4 mysql:latest ip-172-31-37-135.ec2. internal 
Running Running 26 seconds ago 

6jtkvstssnkf mysql.5 mysql:latest ^ ip-172-31-10-132.ec2.internal 
Running Running 26 seconds ago 


Only one mysql service replica is running on the manager node; therefore, only one Docker container 
for the mysql service is running on the manager node. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND 

CREATED STATUS PORTS NAMES 

6bbe40000874 mysql:latest "docker-entrypoint..." 

About a minute ago Up About a minute  3306/tcp mysql.2.s4flvtode80dj;jere2z 


si9gdx 
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Scaling to 10 tasks starts multiple tasks on each of the Swarm nodes. 


" $ docker service scale mysql-10 
mysql scaled to 10 
~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
s4flvtode8od mysql.2 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running about a minute ago 

j0jd92p5dmd8 mysql .3 mysql: latest ip-172-31-10-132.ec2.internal 
Running Running 2 minutes ago 

6jtkvstssnkf mysql.5 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running about a minute ago 

jxunbdec3fnj mysql.6 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 14 seconds ago 

tinz59dyoi2s mysql.7 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 14 seconds ago 

lousvchdirn9 mysql.8 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 14 seconds ago 

94m10f52344d mysql.9 mysgl:latest ip-172-31-37-135.ec2.internal 
Running Running 14 seconds ago 

pd40sd7qlk3j mysql.10 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 14 seconds ago 


The number of Docker containers for the mysql service on the manager node increases to three for the 
three tasks running on the manager node. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 
STATUS PORTS NAMES 
15e3253f69f1 mysql:latest "docker-entrypoint..." | 50 seconds ago 


Up 49 seconds 3306/tcp mysql.8.lousvchdirn9fv8wot5vivk6d 

cca7ab20c914 mysql: latest "docker-entrypoint..." 50 seconds ago 

Up 49 seconds 3306/tcp mysql.10.pd40sd7qlk3jc0i73huop8e4r 
6bbe40000874 mysql: latest "docker-entrypoint..." 2 minutes ago 

Up 2 minutes 3306/tcp mysql.2.s4flvtode8odjjere2zsi9gdx 


Because you'll learn more about Docker services with the MySQL database service example in later 
sections, and also for completeness, next we discuss using a Docker container for MySQL database to create 
a database table. 


Accessing a MySQL Database in a Docker Container 


Next, we access MySQL database in a Docker container. The docker ps command, when run on each 
instance, lists Docker containers for the mysql service on the instance. Start a bash shell for a Docker 
container with the docker exec -it «containerid» bash command. The root prompt gets displayed for 
the Docker container. 


~ $ docker exec -it 15e3253f69f1 bash 
root@15e3253f69T1: /# 
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Start the MySQL CLI with the mysql command as user root. Specify the password when prompted; 
the password used to create the service was specified in the - -env option to the docker service create 
command using environment variable MYSOL ROOT PASSWORD. The mysql» CLI command prompt is 
displayed. 


root@15e3253f69f1:/# mysql -u root -p 

Enter password: 

Welcome to the MySQL monitor. Commands end with ; or Mg. 

Your MySOL connection id is 4 

Server version: 5.7.19 MySOL Community Server (GPL) 

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. 
Oracle is a registered trademark of Oracle Corporation and/or its 
affiliates. Other names may be trademarks of their respective 

owners. 

Type 'help;' or '\h' for help. Type 'Nc' to clear the current input statement. 
mysql» 


Set the database to use as mysql with the use mysql command. 
mysql» use mysql; 
Reading table information for completion of table and column names 
You can turn off this feature to get a quicker startup with -A 
Database changed 


Create a database table with the following SQL script. 


CREATE TABLE wlslog(time stamp VARCHAR(45) PRIMARY KEY,category VARCHAR(25),type 
VARCHAR(25),servername VARCHAR(25),code VARCHAR(25),msg VARCHAR(45)); 


The wlslog table is created. 
mysql» CREATE TABLE wlslog(time stamp VARCHAR(45) PRIMARY KEY,category VARCHAR(25),type 
VARCHAR(25),servername VARCHAR(25),code VARCHAR(25),msg VARCHAR(45)); 

Ouery OK, O rows affected (0.06 sec) 

Add some data to the wlslog table with the following SQL commands run from the MySQL CLI. 
mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:16-PM-PDT' , 'Notice', 'WebLogicServer', 
'AdminServer','BEA-000365','Server state changed to STANDBY'); 

Query OK, 1 row affected (0.02 sec) 
mysql» INSERT INTO wlslog VALUES('Apr-8-2014-7:06:17-PM-PDT' , 'Notice', 'WebLogicServer', 


'AdminServer','BEA-000365','Server state changed to STARTING'); 
Query OK, 1 row affected (0.01 sec) 
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mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:18-PM-PDT','Notice','WebLogicServer', 
'AdminServer','BEA-000365','Server state changed to ADMIN'); 
Query OK, 1 row affected (0.00 sec) 


mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:19-PM-PDT' , 'Notice', 'WebLogicServer', 
'AdminServer','BEA-000365','Server state changed to RESUMING'); 
Query OK, 1 row affected (0.00 sec) 


mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:20-PM-PDT' , 'Notice', 'WebLogicServer', 
'AdminServer','BEA-000331','Started WebLogic AdminServer'); 
Query OK, 1 row affected (0.01 sec) 


mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:21-PM-PDT' , 'Notice', 'WebLogicServer', 
'AdminServer','BEA-000365','Server state changed to RUNNING'); 
Query OK, 1 row affected (0.00 sec) 


mysql> INSERT INTO wlslog VALUES('Apr-8-2014-7:06:22-PM-PDT' , 'Notice', 'WebLogicServer', 
'AdminServer','BEA-000360','Server started in RUNNING mode'); 
Query OK, 1 row affected (0.00 sec) 


Run a SQL query to list the database table data. 


mysql» SELECT * FROM wlslog; 


| Apr-8-2014-7:06:16-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000365 | Server state changed to STANDBY | 
| Apr-8-2014-7:06:17-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000365 | Server state changed to STARTING| 
| Apr-8-2014-7:06:18-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000365 | Server state changed to ADMIN | 
| Apr-8-2014-7:06:19-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000365 | Server state changed to RESUMING| 
| Apr-8-2014-7:06:20-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000331 | Started WebLogic AdminServer | 
| Apr-8-2014-7:06:21-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000365 | Server state changed to RUNNING | 
| Apr-8-2014-7:06:22-PM-PDT | Notice | WebLogicServer | AdminServer | BEA-000360 | Server started in RUNNING mode | 
+--------------------------- +---------- +---------------- +------------- +------------ +--------------------------------- + 
7 rows in set (0.00 sec) 


Exit the MySQL CLI and the bash shell using the exit command. 
mysql> exit 
Bye 


root@15e3253f69f1:/# exit 
exit 
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Updating a Service 


A service may be updated subsequent to being created with the docker service update command, which 
has the following syntax: 


docker service update [OPTIONS] SERVICE 


Some of the supported options are listed in Table 4-4. 


Table 4-4. Options for the docker service update Command 
Option Description 


--args Args for the command. 
--constraint-add Adds or updates a placement constraint. 
--constraint-rm Removes a placement constraint. 

- -container-label-add Adds or updates a Docker container label. 


--container-label-rm Removes a container label by its key. 


--env-add Adds or updates an environment variable. 

--env-rm Removes an environment variable. 

--force Whether to force an update even if no changes require it. Option added in 
Docker 1.13. Default is false. 

--group-add Adds an additional supplementary user group to the container. Option 
added in Docker 1.13. 

--group-rm Removes a previously added supplementary user group from the 
container. Option added in Docker 1.13. 

- -host - add Adds or updates a custom host-to-IP mapping (host:ip). Option added in 
Docker 1.13. 

--host-rm Removes a custom host-to-IP mapping (host: ip). Option added in 
Docker 1.13. 

- -hostname Updates the container hostname. Option added in Docker 1.13. 

--image Updates the service image tag. 

- -Jabel-add Adds or updates a service label. 

- -Jabel-rm Removes a label by its key. 

--limit-cpu Updates the limit CPUs. Default value is 0.000. 


- -limit-memory 


Updates the limit memory. Default value is 0. 


- -log-driver Updates logging driver for service. 
--log-opt Updates logging driver options. 

- -mount - add Adds or updates a mount on a service. 
--mount-rm Removes a mount by its target path. 

- -publish-add Adds or updates a published port. 

- -publish-rm Removes a published port by its target port. 


(continued) 
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Table 4-4. (continued) 
Option 


--read-only 


--replicas 
--reserve-cpu 
--reserve-memory 
--restart-condition 
--restart-delay 
--restart-max-attempts 
--rollback 


esty. -t 


- -update-delay 
- -update-failure-action 


--update-monitor 
--update-parallelism 


--user, -u 


--workdir, -w 


Description 

Mounts the container's root filesystem as read only. Option added in 
Docker 17.06. Default is false. 

Updates the number of tasks. 

Updates the reserve CPUs. Default is 0.000. 

Updates the reserve memory. Default is 0. 

Updates the restart when condition is met (none, on-failure, or any). 
Updates the delay between restart attempts (ns|us|ms|s|m|h). 
Updates the maximum number of restarts before giving up. 


Whether to roll back to a previous specification. Option added in Docker 
1.13. Default is false. 


Whether to allocate a pseudo-TTY. Option added in Docker 1.13. 
Default is false. 


Updates delay between updates (ns|us|ms|s|m|h). Default is Os. 
Updates action on update failure (pause|continue). Default is pause. 


Duration after each task update to monitor for failure (ns|us|ms|s|m]|h). 
Option added in Docker 1.13. Default Os. 


Updates the maximum number of tasks updated simultaneously 
(0 to update all at once). Default is 1. 


Adds the username or UID (format: «name |uid»[ : «group|gid» ]). 


Updates the working directory inside the container. 


Next, we update some of the parameters of a deployed service. 


Updating the Replicas 


First, create a mysql service to update. 


docker service create \ 


--env MYSOL ROOT PASSWORD=' mysql’ \ 


--replicas 1 \ 


--restart-condition on-failure \ 
--restart-max-attempts 5 \ 
--update-failure-action continue \ 


--name mysql \ 
--update-delay 10s \ 
mysql:5.6 
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A service from Docker image mysq1:5.6 is created and the service ID is output. 


~ $ docker service rm mysql 

mysql 

~ $ docker service create \ 
--env MYSQL ROOT PASSWORD= 'mysql'N 
--replicas 1 \ 
--restart-condition on-failure \ 
--restart-max-attempts 5 \ 
--update-failure-action continue \ 
--name mysql \ 
--update-delay 10s \ 

mysql:5.6 

mecdt3zluvlvxqc3hdpw8edg1 


> 
> 
> 
> 
> 
> 
> 
> 


Update the number of replicas to five using the docker service update command. If the command is 
successful, the service name is output from the command. 


~ $ docker service update --replicas 5 mysql 
mysql 


Setting replicas to five does not just start four new tasks to make a total of five tasks. When a service 
is updated to change the number of replicas, all the service tasks are shut down and new tasks are started. 
Subsequently listing the service tasks lists the first task as being shut down and five new tasks as being 
started. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
jenofmk;jj13k mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Running Starting less than a second ago 

r616gx5880pd V mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 5 seconds ago "task: non-zero exit (137)" 
y350n4e8furo mysql.2 mysql:5.6 ip-172-31-13-155.ec2.internal 
Running Running 7 seconds ago 

ktrwxnn13fug mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Running Running 14 seconds ago 

2t8j1zd8uts1 mysql.4 mysql:5.6 ip-172-31-10-132.ec2.internal 
Running Running 10 seconds ago 

8tfOuuwb8i31 mysql.5 mysql:5.6 ip-172-31-10-132.ec2.internal 
Running Running 10 seconds ago 


Updating the Docker Image Tag 


Starting with a MySQL database service called mysql for Docker image mysq1:5.6, next we update the 
service to a different Docker image tag—the mysql: latest Docker image. Run the following command to 
update the Docker image; the service name is output to indicate that the update is successful. 


~ $ docker service update --image mysql:latest mysql 
mysql 
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You can list detailed information about the service with the docker service inspect command. The 
image listed in the ContainerSpec is mysql: latest. The PreviousSpec is also listed. 


~ $ docker service inspect mysql 


[ 
{ 
"Spec": { 

"Name": "mysql", 

"Labels": {}, 

"TaskTemplate": { 

"ContainerSpec": { 
"Image": “mysql: latest@sha256:75c563c474f1adc149978011fedfe2e6670483d133 
b22b07ee32789b626f8de3", 
"Env": | 
"MYSQL ROOT PASSWORD=mysql" 
|, 
"PreviousSpec": { 

"Name": "mysql", 

"Labels": {}, 

"TaskTemplate": 1 

"ContainerSpec": { 
"Image": "mysql:5.68sha256:6ad5bd392c9190fa92e65fd21f6debc8b2a76fc54f139 
49f9b5bc6a0096a5285", 
] 


The update does not get completed immediately even though the docker service update command 
does. While the service is being updated, the UpdateStatus for the service is listed with State set to 
"updating" and the Message of "update in progress". 


"UpdateStatus": { 
"State": "updating", 
"StartedAt": "2017-07-23T19:24:15.539042747Z" , 
"Message": "update in progress" 


When the update completes, the UpdateStatus State becomes "completed" and the Message becomes 
"update completed". 


"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-23T19:24:15.539042747Z" , 
"CompletedAt": "2017-07-23T19:25:25.660907984Z", 
"Message": "update completed" 
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While the service is updating, the service tasks are shutting down and the new service tasks are starting. 
When the update is starting, some of the running tasks might be based on the previous image mysq1:5.6 
whereas others could be based on the new image mysql: latest. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
jenofmk;jj13k mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Running Running 38 seconds ago 

r616gx5880pd V mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 43 seconds ago "task: non-zero exit (137)" 

y350n4e8furo mysql.2 mysql:5.6 ip-172-31-13-155.ec2.internal 
Running Running 45 seconds ago 

bswz4sm8e3vj mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Running Running 6 seconds ago 

ktrwxnn13fug V mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 12 seconds ago "task: non-zero exit (1)" 

wjix26wvpopt mysql.4 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 7 seconds ago 

2t8j1zd8uts1 V mysql.4 mysql:5.6 ip-172-31-10-132.ec2.internal 
Shutdown Shutdown 7 seconds ago 

hppq840ekrh7 mysql.5 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 2 seconds ago 

8tfOuuwb8i31 V mysql.5 mysql:5.6 ip-172-31-10-132.ec2.internal 
Shutdown Failed 8 seconds ago "task: non-zero exit (1)" 


The desired state of the tasks with image mysq1:5.6 is set to Shutdown. Gradually, all the new service 
tasks based on the new image mysql: latest are started. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
2uafxtcbj9qj mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 30 seconds ago 

jenofmkjj13k V mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 36 seconds ago "task: non-zero exit (137)" 

r616gx5880pd V mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed about a minute ago “task: non-zero exit (137)" 

mkv95bvx3s11 mysql.2 mysql:latest ip-172-31-13-155.ec2.internal 
Ready Ready 3 seconds ago 

y350n4e8furo V mysql.2 mysql:5.6 ip-172-31-13-155.ec2.internal 
Shutdown Failed 4 seconds ago "task: non-zero exit (137)" 

yevunzeri2vm mysql.3 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 12 seconds ago 

bswz4sm8e3vj V mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Shutdown 12 seconds ago 

ktrwxnn13fug V mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 48 seconds ago "task: non-zero exit (1)" 

wjix26wvpopt mysql.4 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 44 seconds ago 


T 


CHAPTER 4 ~ DOCKER SERVICES 


2t8j1zd8uts1 
Shutdown 
hppq840ekrh7 
Running 
8tfOuuwb8i31 
Shutdown 


V mysql.4 mysql:5.6 
Shutdown 44 seconds ago 
mysql.5 
Running 39 seconds ago 

V mysql.5 mysql:5.6 
Failed 44 seconds ago 


mysql:latest 


ip-172-31-10-132.ec2.internal 
ip-172-31-10-132.ec2.internal 


ip-172-31-10-132.ec2.internal 


"task: non-zero exit (1)" 


Filtering the service tasks with the -f option was introduced earlier. To find which, if any, tasks are 
scheduled on a particular node, you run the docker service ps command with the filter set to the node. 


Filtered tasks, both Running and Shutdown, are then listed. 


~ $ docker service ps 


ID 


DESIRED STATE 


mkv95bvx3sl1 
Running 
y350n4e8furo 
Shutdown 
oksssg7gsh79 
Running 
wjix26wvpopt 
Shutdown 


NAME IMAGE 
CURRENT STATE 
mysql.2 mysql:latest 


Running about a minute ago 
V mysql.2 mysql:5.6 
Failed about a minute ago 
mysql.4 
Running 50 seconds ago 
V mysql.4 
Failed 55 seconds ago 


ERROR 


mysql:latest 


mysql:latest 


-f node-ip-172-31-13-155.ec2.internal mysql 


NODE 
PORTS 
ip-172-31-13-155.ec2.internal 


ip-172-31-13-155.ec2.internal 
"task: non-zero exit (137)" 
ip-172-31-13-155.ec2.internal 


ip-172-31-13-155.ec2.internal 
"task: non-zero exit (1)" 


Service tasks may also be filtered by desired state. To list only running tasks, set the desired-state filter 
to running. 


~ $ docker service ps -f desired-state-running mysql 
ID NAME IMAGE NODE 


DESIRED STATE CURRENT STATE ERROR PORTS 


2uafxtcbj9qj mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 3 minutes ago 
mkv95bvx3s11 mysql.2 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 2 minutes ago 
yevunzeri2vm mysql.3 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 2 minutes ago 
oksssg7gsh79 mysql.4 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 2 minutes ago 
hppq840ekrh7 mysql.5 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 3 minutes ago 


Likewise, only the shutdown tasks are listed by setting the desired-state filter to shutdown. 


~ $ docker service ps -f desired-state-shutdown mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
jenofmkjj13k mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 3 minutes ago "task: non-zero exit (137)" 

r616gx5880pd V mysql.1 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 3 minutes ago "task: non-zero exit (137)" 

y350n4e8furo mysql.2 mysql:5.6 ip-172-31-13-155.ec2.internal 
Shutdown Failed 2 minutes ago "task: non-zero exit (137)" 
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bswz4sm8e3vj mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Shutdown 2 minutes ago 

ktrwxnn13fug V mysql.3 mysql:5.6 ip-172-31-37-135.ec2.internal 
Shutdown Failed 3 minutes ago "task: non-zero exit (1)" 

wjix26wvpopt mysql.4 mysql:latest ip-172-31-13-155.ec2.internal 
Shutdown Failed 2 minutes ago "task: non-zero exit (1)" 

2t8j1zd8uts1 V mysql.4 mysql:5.6 ip-172-31-10-132.ec2.internal 
Shutdown Shutdown 3 minutes ago 

8tfOuuwb8i31 mysql.5 mysql:5.6 ip-172-31-10-132.ec2.internal 
Shutdown Failed 3 minutes ago "task: non-zero exit (1)" 


Updating the Placement Constraints 


The placement constraints may be added/removed with the - -constraint-add and --constraint-rm 
options. We started with a Swarm consisting of three nodes—one manager and two worker nodes. We then 
promoted a worker node to a manager, resulting in a Swarm with two manager nodes and one worker node. . 

Starting with service replicas running across the Swarm nodes, the replicas may be constrained to run 
on only worker nodes with the following command. The docker service update command outputs the 
service name if successful. 


~ $ docker service update --constraint-add 
mysql 


"node.role--worker" mysql 


It may take a while (a few seconds or minutes) for the desired state of a service to be reconciled, during 
which time tasks could be running on manager nodes even though the node.role is set to worker or less 
than the required number of tasks could be running. When the update has completed (the update status 
may be found from the docker service inspect command), listing the running tasks for the mysql service 
indicates that the tasks are running only on the worker nodes. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

smk5q4nhu1rw mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running about a minute ago 

wzmou8f6r2tg mysql.2 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 23 seconds ago 

byavev89hukv mysql.3 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 23 seconds ago 

erx409p0sgcc mysql.4 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 53 seconds ago 

q7equ8jlqig8 mysql.5 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 46 seconds ago 


As another example, service tasks for the mysql service may be constrained to run on only manager 


nodes. Starting with service tasks running on both manager and worker nodes and with no other constraints 


added, run the following command to place all tasks on the manager nodes. 


~ $ docker service update --constraint-add 'node.role--manager' mysql 


mysql 
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The tasks are not shut down on worker nodes and started on manager nodes immediately and initially 
may continue to be running on worker nodes. 

List the service replicas again after a while. You'll see that all the tasks are listed as running on the 
manager nodes. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

7tj8bck4jr5n mysql.1 mysql: latest ip-172-31-13-155.ec2.internal 
Running Running 14 seconds ago 

uyeu3y67v2rt mysql.2 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running about a minute ago 

1t9p7479lkta mysql.3 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 1 second ago 

t7d9c4viuo5y mysql.4 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 40 seconds ago 

8xufz871yx1x mysql.5 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 27 seconds ago 


Updating Environment Variables 


The --env-add and --env-rm options are used to add/remove environment variables to/from a service. The 
mysql service we created includes only one environment variable—the mandatory MYSQL ROOT PASSWORD 
variable. You can use the docker service update command to add the environment variables MYSQL -. 
DATABASE, MYSOL PASSWORD, and MYSOL ALLOW EMPTY PASSWORD and to update MYSOL ROOT PASSWORD in the 
same command to an empty password. The command outputs the service name if successful. 


~ $ docker service update --env-add ‘MYSQL DATABASE-mysql'  --env-add 'MYSOL 
PASSWORD-mysql' --env-add 'MYSOL ALLOW EMPTY PASSWORD-yes' --env-add ‘MYSQL ROOT _ 
PASSWORD-yes' mysql 

mysql 


When the update has completed, the docker service inspect command lists the environment 
variables added. 


~ $ docker service inspect mysql 
[sss 
"Spec": { 
"Name": "mysql", 


"Env": [ 
"MYSOL ROOT PASSWORD-yes", 
"MYSQL  DATABASE-mysql" , 
"MYSOL PASSWORD-mysql", 
"MYSOL ALLOW EMPTY PASSWORD-yes" 


Updating the environment variables causes the containers to restart. So, simply adding environment 
variables doesn't cause the new database to be created in the same container. A new container is started 
with the updated environment variables. 
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Updating the Docker Image 


The Docker image may also be updated, not just the image tag. As an example, update the Docker image 
for a MySQL database service to use the postgres Docker image, which is for the PostgreSQL database. 
The command outputs the service name if the update is successful. 


~ $ docker service update --image postgres mysql 
mysql 


After the update has completed, showing the running service tasks lists new tasks for the postgres 
image. The service name stays the same and the Docker image is updated to postgres. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

hmk71281s19a mysql.1 postgres:latest ip-172-31-13-155.ec2.internal 
Running Running 18 seconds ago 

5ofbkc82gpoi mysql.2 postgres:latest ip-172-31-10-132.ec2.internal 
Running Running about a minute ago 

vogfc65lhw62 mysql.3 postgres:latest ip-172-31-13-155.ec2.internal 
Running Running 31 seconds ago 

miscjf9n66qq mysql.4 postgres:latest ip-172-31-13-155.ec2.internal 
Running Running 45 seconds ago 

g5viy8jyzpi1 mysql.5 postgres:latest ip-172-31-10-132.ec2.internal 
Running Running about a minute ago 


Updating the Docker image does not remove the environment variables associated with the mysql 
Docker image, which are still listed in the service detail. 


~ $ docker service inspect mysql 


"Spec": { 
"Name": "mysql", 


"ContainerSpec": { 
"Env": [ 
"MYSQL ROOT PASSWORD-yes", 
"MYSQL _DATABASE=mysql", 
"MYSOL PASSWORD-mysql", 
"MYSOL ALLOW EMPTY PASSWORD-yes" 
|, 


The added environment variables for the MySQL database need to be removed, as the PostgreSQL 
database Docker image postgres does not use the same environment variables. Remove all the environment 
variables from the mysql service with the --env-rm option to the docker service update command. 

To remove only the env variable, the name needs to be specified, not the env value. 


docker service update --env-rm ‘MYSQL DATABASE’ --env-rm ‘MYSQL PASSWORD’ --env-rm 
'MYSOL ALLOW EMPTY PASSWORD'  --env-rm 'MYSOL ROOT PASSWORD' mysql 


8l 


CHAPTER 4 ~ DOCKER SERVICES 


Updating the Container Labels 


The --container-label-add and --container-label-rm options are used to update the Docker container 
labels for a service. To add a container label to the mysql service, run a docker service update command, 
which outputs the service name if successful. 


~ $ docker service update --container-label-add 'com.docker.swarm.service.version-latest' 
mysql 
mysql 


On listing detailed information about the service, the added label is listed in the ContainerSpec labels. 


~ $ docker service inspect mysql 


"ContainerSpec": { 
"Labels": { 
"com.docker.swarm.service.version": "latest" 
js 


The label added may be removed with the - -container-label-rm option. To remove only the label, the 
key needs to be specified, not the label value. 


~ $ docker service update --container-label-rm  'com.docker.swarm.service.version' mysql 
mysql 


Updating Resources Settings 


The - -limit-cpu, --limit-memory, --reserve-cpu, and --reserve-memory options of the docker service 
update command are used to update the resource settings for a service. As an example, update the resource 
limits and reserves. The command outputs the service name if successful. 


~ $ docker service update --limit-cpu 0.5 --limit-memory 1GB --reserve-cpu 
"0.5" --reserve-memory "1GB" mysql 
mysql 


The resources settings are updated. Service detail lists the updated resource settings in the Resources 
JSON object. 


~ $ docker service inspect mysql 


"ContainerSpec": { 
"Resources": { 
"Limits": { 
"NanoCPUs": 500000000, 
"MemoryBytes": 1073741824 


3 
"Reservations": { 


82 


CHAPTER 4... DOCKER SERVICES 


"NanoCPUs": 500000000, 
"MemoryBytes": 1073741824 


J, 


Removing a Service 


The docker service rmcommand removes a service. If the output of the command is the service name, the 
service has been removed. All the associated service tasks and Docker containers also are removed. 


~ $ docker service rm mysql 
mysql 


Creating a Global Service 


As discussed earlier, a service has two modes—replicated or global. The default mode is replicated. 

The mode may also be explicitly set to replicated with the - -mode option of the docker service create 
command. The service mode cannot be updated after a service has been created, with the docker service 
update command for example. Create a replicated service for nginx using the - -mode option. 


~ $ docker service create --mode replicated --name nginx nginx 
no177eh3gxsyemb1gfzc99mmd 


A replicated mode service is created with the default number of replicas, which is 1. List the services 
with the docker service ls command. The nginx service is listed with one replica. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
no177eh3gxsy nginx replicated 1/1 nginx:latest 


A global service runs one task on each node in a Swarm by default. A global service may be required 
at times such as for an agent (logging/monitoring) that needs to run on each node. A global service is used 
for logging in Chapter 11. Next, we create a nginx Docker image-based service that's global. Remove the 
replicated service nginx with the docker service rm nginx command. A service name must be unique 
even if different services are of different modes. Next, create a global mode nginx service with the same 
command as for the replicated service, except that the - -mode option is set to global instead of replicated. 


~ $ docker service create --mode global --name nginx nginx 
5prj6c4v4be6ga0odnb22qa4n 


A global mode service is created. The docker service ls command lists the service. The REPLICAS 
column for a global service does not list the number of replicas, as no replicas are created. Instead global is 
listed in the REPLICAS column. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
5prj6c4v4be6 nginx global 3/3 nginx:latest 
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A service task is created for a global service on each node in the Swarm on which a task can run. 
Scheduling constraints may be used with a global service to prevent running a task on each node. 
Scheduling is discussed in Chapter 8. Global services cannot be scaled. 


Summary 


This chapter introduced Docker services running on a Docker Swarm. A service consists of service tasks or 
replicas. A Docker Swarm supports two types of services—replicated services and global services. A replicated 
service has the assigned number of replicas and is scalable. A global service has a task on each node in a 
Swarm. The term “replica” is used in the context of a replicated service to refer to the service tasks that are 
run across the nodes in a Swarm. A replicated service could run a specified number of tasks for a service, 
which could imply running no tasks or running multiple tasks on a particular node. The term “replica” is 
generally not used in the context of a global service, which runs only one task on each node in the Swarm. 
Each task (replica) is associated with a Docker container. We started with a Hello World service and invoked 
the service with curl on the command line and in a browser. Subsequently, we discussed a service for 

a MySQL database. We started a bash shell for a MySQL service container and created a database table. 
Scaling, updating, and removing a service are some of the other service features this chapter covered. 

The chapter concluded by creating a global service. The next chapter covers the Docker Swarm scaling 
service in more detail. 
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CHAPTER 5 


Scaling Services 


Docker Engine is suitable for developing lightweight applications that run in Docker containers that are 
isolated from each other. Docker containers are able to provide their own networking and filesystem. 


The Problem 


Docker Engine (prior to native Swarm mode) was designed to run Docker containers that must be started 
separately. Consider the use case that multiple replicas or instances of a service need to be created. 

As client load on an application running in a Docker container increases, the application may need to be 
run on multiple nodes. A limitation of Docker Engine is that the docker run command must be run each 
time a Docker container is to be started for a Docker image. If a Docker application must run on three nodes, 
the docker run <img> command must run on each of the nodes as well, as illustrated in Figure 5-1. 

No provision to scale an application or run multiple replicas is provided in the Docker Engine 

(prior to Docker 1.12 native Swarm mode support). 


run run run 
<img> <img> <img> 





Node Node Node 


Figure 5-1. Docker engine without provision for scaling 
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The Solution 


The Docker Swarm mode has the provision to scale a Docker service. A service abstraction is associated 
with zero or more replicas (tasks) and each task starts a Docker container for the service. The service 
may be scaled up or down to run more/fewer replicas, as required. With a single docker service 

scale «svc»-«replicas» command, a service can run the required number of replicas, as illustrated in 
Figure 5-2. If 10 service replicas are to be started across a distributed cluster, a single command is able to 
provision scaling. 








Docker 


docker service scale containers 


<svc>=<number of 
tasks» 





Node 
3 


Figure 5-2. Docker Swarm mode with provision for scaling 


Scaling is supported only for replicated services. A global service runs one service task on each node 
in a Swarm. Scaling services was introduced in Chapter 3 and, in this chapter, we discuss some of the other 
aspects of scaling services not discussed in Chapter 3. This chapter covers the following topics: 


e Setting the environment 

e Creating a replicated service 

e Scaling up a service 

e Scaling down a service 

e Removing a service 

e Global services cannot be scaled 

e Scaling multiple services in the same command 


e Service replicas replacement on a node leaving the Swarm 
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Setting the Environment 


Create a three-node Swarm on Docker for Swarm, which is discussed in Chapter 3. A Docker for AWS Swarm 
you created in another chapter may be used in this chapter. Obtain the public IP address of the EC2 instance 
for the Swarm manager. 

SSH login to the Swarm manager EC2 instance with user “docker” 


[root@localhost ~]# ssh -i "docker.pem"  docker834.200.225.39 
Welcome to Docker! 


The docker node ls command lists the nodes in the Swarm. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
ilru4f0i280w2tlsrg9hglwsj ^ ip-172-31-10-132.ec2.internal Ready Active 
w5toi186ipblpcq390625wyq2e ip-172-31-37-135.ec2.internal Ready Active 
zkxle7kafwcmtisd93kh5cy5e * ip-172-31-13-155.ec2.internal Ready Active Leader 


Creating a Replicated Service 


As discussed in Chapter 4, Docker Swarm mode supports two types of services—global and replicated. The 

default is the replicated mode. Only the replicated service can be scaled. Next, create a replicated service for 
MySQL database using the docker service create command, initially consisting of one replica, as specified 
in the - - replicas option. The default number of replicas if the - - replicas option is not specified is also one. 


~ $ docker service create ^ 
--env MYSQL ROOT PASSWORD- 'mysql'N 
--replicas 1 \ 
--name mysql ^ 

mysql 

ndu4kwqk9017e7wxvv5bremr4 


> 
> 
> 
> 


List the services using docker service ls. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
ndu4kwqk9017 mysql replicated 1/1 mysql: latest 


As service replicas take a while (albeit a few seconds) to start, initially 0/1 replicas could be listed in the 
REPLICAS column, which implies that the desired state of running one service replica has not been achieved 
yet. Run the same command after a few seconds and 1/1 REPLICAS should be listed as running. 

Optionally, the docker service create command may also be run by setting the - -mode option. 
Remove the mysql service if it was created previously and use the - -mode option as follows. 


~ $ docker service rm mysql 
mysql 
~ $ docker service create \ 


> --mode replicated \ 

>  --env MYSQL ROOT PASSWORD='mysql' \ 
>  --replicas 1 ^ 

>  --name mysql ^ 

» mysql 


rl2s2ptgbs9z2t7fy5e63wf2j 
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The mysql service is created as without the - -mode replicated option. List the service replicas or tasks 
with docker service ps mysql.A single replica is listed. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 
yrikmh7mciv7 mysql.1 mysql: ip-172-31-13- Running Running 21 
latest 155.ec2.internal seconds ago 


One service replica is created by default if the - - replicas option is omitted. It should be mentioned 
that running multiple replicas of the MySQL database does not automatically imply that they are sharing 
data, so accessing one replica will not give you the same data as another replica. Sharing data using mounts 
is discussed in Chapter 6. 


Scaling Up a Service 


Thedocker service scale command, which has the following syntax, may be used to scale up/down a 
service, which changes the desired state of the service. 


docker service scale SERVICE-REPLICAS [SERVICE=REPLICAS... | 
First, scale up the service to three replicas. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


Subsequently, three tasks are listed as scheduled on the three nodes in the Swarm. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 
yrikmh7mciv7 mysql.1 mysql: ip-172-31-13- Running Running 37 
latest 155.ec2.internal seconds ago 
3zxmotmy6n2t mysql.2 mysql: ip-172-31-37- Running Running 7 
latest 135.ec2.internal seconds ago 
rdfsowttd3b9 mysql.3 mysql: ip-172-31-10- Running Running 7 
latest 132.ec2.internal seconds ago 


In addition to one replica on the manager node, one replica each is started on each of the two worker 
nodes. If the docker ps command is run on the manager node, only one Docker container for the mysql 
Docker image is listed. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 
6d2161a3b282 mysql:  "docker- 50 seconds ago Up 49 3306/tcp mysql.1.yrikmh7mci 
latest entrypoint..." seconds v7dsmqlinhdi621 
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A service may also be scaled using the docker service update command with the --replicas option. 


As an example, scale it to 50 replicas. 


~ $ docker service update --replicas-50 mysql 


mysql 


The service is scaled to 50 replicas and, subsequently, 50 service tasks are listed. 


~ $ docker service ps -f desired-state-running mysql 


ID 


DESIRED STATE 


to26kjbsgzmq 
Running 
f3tx2kbe55dh 
Running 
5mzej75us115 
Running 
wluix1b3z863 
Running 
9ld8smvahk9g 
Running 
3tgw8nibsmfii 
Running 
1gm8e7pxkgoo 
Running 
iq5p2g480agq 
Running 
i4yho72h1gs6 
Running 
riz5tgu0dg13 
Running 
mekfjvxi9pds 
Running 
nd8f2pr4oivc 
Running 
xou9hzt1j637 
Running 
t95flokvca2y 
Running 
rda5shwwfmsc 
Running 
ibb2fk211m3w 
Running 
st4ofpvrfaip 
Running 
iw4daunt6s63 
Running 
vk4nzq7uty12 
Running 
0j59qjcy51qw 
Running 


NAME 


CURRENT STATE 


mysql.1 


IMAGE 


mysql: 


Running 11 seconds ago 


mysql.2 


mysql: 


Running 20 seconds ago 


mysql.3 


mysql: 


Running 13 seconds ago 


mysql.4 


mysql: 


Preparing 13 seconds ago 


mysql.5 


mysql: 


Running 47 seconds ago 


mysql.6 
Running 46 
mysql.7 
Running 46 
mysql.8 
Running 45 
mysql.9 
Running 46 
mysql.10 
Running 45 
mysql.11 
Running 46 
mysql.12 
Running 45 
mysql.13 
Running 45 
mysql.14 
Running 45 
mysql.15 
Running 45 
mysql.16 
Running 47 
mysql.17 
Running 45 
mysql.18 
Running 47 
mysql.19 
Running 46 
mysql.20 
Running 45 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


ERROR 
latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


NODE 
PORTS 


ip-172-31-37-135. 
ip-172-31-10-132. 
ip-172-31-10-132. 
ip-172-31-13-155. 
ip-172-31-13-155. 
ip-172-31-10-132. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-13-155. 
ip-172-31-10-132. 
ip-172-31-13-155. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-10-132. 


ip-172-31-37-135. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2 


Bc2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2. 


ec2 


ec2. 


ec2. 


Bc2. 


ec2. 


internal 


internal 


internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


internal 


internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


internal 
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wiou769z8xeh 
Running 
5exwimn64w94 
Running 
agqongnh9uu3 
Running 
ynkvjwgqqq1x 
Running 
yf87kbsn1cga 
Running 
xxqj62007cxd 
Running 
50ym9i8tjwd5 
Running 
7btl2pga1150 
Running 
62dqj60q1018 
Running 
psn7zl4th2zb 
Running 
khsj2an2f5gk 
Running 
rzpndzjpmuj7 
Running 
9zrcga93u5fi 
Running 
x565ry5ugj8m 
Running 
010s5dievj37 
Running 
dritgxqozrua 
Running 
n8hsO1m8picr 
Running 
dk5woqnkfb63 
Running 
joiii03na4ao 
Running 
db5hz7m2vac1 
Running 
ghk6s12eeo48 
Running 
jbi8aksksozs 
Running 
rx3rded300a4 
Running 
c3zaacke440s 
Running 
l6ppiurx4306 
Running 
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mysql.21 
Running 47 
mysql.22 
Running 48 
mysql.23 
Running 45 
mysql.24 
Running 47 
mysql.25 
Running 10 
mysql.26 
Running 45 
mysql.27 
Running 45 
mysql.28 
Running 46 
mysql.29 
Running 45 
mysql.30 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


Preparing 16 seconds ago 


mysql.31 


mysql: 


Running 45 seconds ago 


mysql.32 
Running 45 
mysql.33 
Running 45 
mysql.34 
Running 48 
mysql.35 
Running 46 
mysql.36 
Running 45 
mysql.37 
Running 47 
mysql.38 
Running 45 
mysql.39 
Running 45 
mysql.40 
Running 46 
mysql.41 
Running 45 
mysql.42 
Running 47 
mysql. 43 
Running 47 
mysql.44 
Running 45 
mysql.46 
Running 46 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


seconds 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


mysql: 


ago 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


latest 


ip-172-31-10-132. 
ip-172-31-10-132. 
ip-172-31-37-135. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-37-135. 
ip-172-31-10-132. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-13-155. 
ip-172-31-10-132. 
ip-172-31-10-132. 
ip-172-31-37-135. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-13-155. 
ip-172-31-37-135. 
ip-172-31-13-155. 


ip-172-31-10-132. 


ec2 


ec2. 


ec2. 


ec2. 


ec2. 


ec2 


ec2. 


ec2. 


ec2 s 


ec2 


ec2. 


ec2. 


ec2. 


ec2. 


ec2 


ec2. 


ec2. 


ec2. 


ec2 


ec2s 


ec2. 


ec2. 


ec2. 


ec2 


ec2. 


.internal 


internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


.internal 


internal 


internal 


internal 


internal 


.internal 


internal 
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of06zibtlsum mysql.47 mysql: latest ip-172-31-10-132.ec2.internal 
Running Running 46 seconds ago 
kgjjwlc9zmp8 mysql. 48 mysql: latest ip-172-31-10-132.ec2.internal 
Running Running 46 seconds ago 
rwiicgkyw61u mysql.49 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 46 seconds ago 
j5jp19a5jgbj mysql.50 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 47 seconds ago 


A small-scale MySQL database service probably wouldn't benefit from scaling to 50 replicas, but an 
enterprise-scale application could use 50 or even more replicas. 


Scaling Down a Service 


A service may be scaled down just as it is scaled up. A service may even be scaled down to no replicas. The 
mysql service may be scaled down to no replicas by setting the number of replicas to 0 using the docker 
service update or docker service scale command. 


~ $ docker service scale mysql=0 
mysql scaled to O 


The service gets scaled down to no replicas. No service replicas that are running are listed. 
~ $ docker service ps -f desired-state-running mysql 
ID NAME IMAGE NODE DESIRED STATE 
CURRENT STATE ERROR PORTS 

The actual service tasks could take a while to shut down, but the desired state of all tasks is set to 
Shutdown. 

Scaling a service to no tasks does not run any tasks, but the service is not removed. The mysql service 


may be scaled back up again from none to three tasks as an example. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


Three service tasks start running. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
py7aqwy2reku mysql.1 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 9 seconds ago 

re113q3iwmvo mysql.2 mysql:latest  ip-172-31-37-135.ec2.internal 
Running Running 9 seconds ago 

h7my2ucpfz3u mysql.3 mysql:latest  ip-172-31-37-135.ec2.internal 
Running Running 9 seconds ago 
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Removing a Service 


A service may be removed using the docker service rm command. 


~ $ docker service rm mysql 
mysql 


The mysql service is not listed after having been removed. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 


Multiple services may be removed using the docker service rmcommand. To demonstrate, you can 
create two services, hello-world and nginx. 


~ $ docker service create \ 

--name hello-world \ 

--publish 8080:80 \ 

--replicas 2 \ 

>  tutum/hello-world 
t3msb25rc8b6xcm30kozoh4ws 

~ $ docker service create --name nginx nginx 
ncn4aqkgzrcjc8wiuorjo5jrd 

~ $ docker service ls 


VV MV 


ID NAME MODE REPLICAS IMAGE PORTS 
ncn4aqkgzrcj nginx replicated 1/1 nginx: latest 
t3msb25rc8b6 hello-world replicated 2/2 tutum/hello-world:latest *:8080->80/tcp 


Subsequently, remove both the services with one docker service rm command. The services removed 
are output if the command is successful. 


~ $ docker service rm nginx hello-world 
nginx 
hello-world 


Global Services Cannot Be Scaled 


A global service creates a service task on each node in the Swarm and cannot be scaled. Create a global 
service for a MySQL database using the docker service create command. Notable differences in the 
command are that the - -mode is set to global and the - -replicas option is not included. 


~ $ docker service create ^ 

--mode global \ 

--env MYSQL ROOT PASSWORD= ‘mysql '\ 
>  --name mysql-global \ 

> mysql 

nxhnrsiulymd9n4171cie938j 


> 
> 
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The global service is created and listing the service should indicate a Mode set to global. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
nxhnrsiulymd mysql-global global 3/3 mysql:latest 


One service task is created on each node in the Swarm. 


~ $ docker service ps mysql-global 


ID NAME IMAGE 

NODE DESIRED STATE CURRENT STATE ERROR PORTS 
nfbmkqdh46ko mysql-global.zkxle7kafwcmtisd93kh5cyse ^ mysql:latest 
ip-172-31-13-155.ec2.internal Running Running 22 seconds ago 

t55ba3bobwzf mysql-global.w5to186ipblpcq390625wyq2e mysql:latest 
ip-172-31-37-135.ec2.internal Running Running 22 seconds ago 

kqg656m301j3 mysql-global.ilru4f0i280w2tlsrg9hglwsj ^ mysql:latest 
ip-172-31-10-132.ec2.internal Running Running 22 seconds ago 


If another node is added to the Swarm, a service task automatically starts on the new node. 
Ifthe docker service scale command is run for the global service, the service does not get scaled. 
Instead, the following message is output. 


~ $ docker service scale mysql-global=5 
mysql-global: scale can only be used with replicated mode 


A global service may be removed just as a replicated service, using the docker service rm command. 


~ $ docker service rm mysql-global 
mysql-global 


scaling Multiple Services Using the Same Command 


Multiple services may be scaled using a single docker service scale command. To demonstrate, create 
two services: nginx and mysql. 


~ $ docker service create \ 


>  --replicas 1 \ 
>  --name nginx ^ 
> nginx 


u6i4e8eg720dwzz425inhxqrp 


~ $ docker service create ^ 

>  --env MYSQL ROOT PASSWORD-'mysql'^ 
>  --name mysql ^ 

» mysql 

1umb7e2gr68s54utujr6khjgd 


List the two services. One replica for each service should be running. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
1umb7e2gr68s mysql replicated 1/1 mysql: latest 
u6i4e8eg720d nginx replicated 1/1 nginx:latest 
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Scale the nginx service and the mysql service with a single command. Different services may be scaled 
to a different number of replicas. 


~ $ docker service scale mysql-5 nginx=10 
mysql scaled to 5 
nginx scaled to 10 


The mysql service gets scaled to five tasks and the nginx service gets scaled to 10 replicas. Initially, some 
of the new tasks for a service may not have started, as for the nginx service, which lists only 8 of the 10 tasks 


as running. 


~ ¢ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
1umb7e2gr68s mysql replicated 5/5 mysql:latest 
u6i4e8eg720d nginx replicated 8/10 nginx:latest 


After a while, all service tasks should be listed as running, as indicated by 10/10 for the nginx service. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
1umb7e2gr68s mysql replicated 5/5 mysql:latest 
u6i4e8eg720d nginx replicated 10/10 nginx:latest 


The service tasks for the two services may be listed using a single docker service ps command. 


~ $ docker service ps nginx mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

f9g1tw88nppk mysql.1 mysql:latest ip-172-31-26-234.ec2.internal 
Running Running about a minute ago 

zcliqfdiqrvu nginx.1 nginx:latest ip-172-31-10-132.ec2.internal 
Running Running about a minute ago 

Vvu4xo99xr0y4 nginx.2 nginx:latest ip-172-31-13-155.ec2.internal 
Running Running 40 seconds ago 

xvxgfoacxjos mysql.2 mysql:latest ip-172-31-37-135.ec2.internal 
Running Running 41 seconds ago 

ywOopq5yOx20 nginx.3 nginx:latest ip-172-31-13-155.ec2.internal 
Running Running 41 seconds ago 

vb92hkua6eyo mysql.3 mysql:latest ip-172-31-13-155.ec2.internal 
Running Running 40 seconds ago 

1cnqwtb2A4zvy nginx.4 nginx:latest ip-172-31-13-155.ec2.internal 
Running Running 41 seconds ago 

hclu53xkosva mysql.4 mysql:latest ip-172-31-26-234.ec2.internal 
Running Running 40 seconds ago 

2xjcw419xw89 nginx.5 nginx: latest ip-172-31-10-132.ec2.internal 
Running Running 41 seconds ago 

ocvb2qctuids mysql.5 mysql:latest ip-172-31-10-132.ec2.internal 
Running Running 41 seconds ago 

18mlu3jpp9cx nginx.6 nginx:latest ip-172-31-10-132.ec2.internal 
Running Running 41 seconds ago 
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p84m8yh5if5t nginx.7 nginx: latest ip-172-31-37-135.ec2.internal 
Running Running 41 seconds ago 
7yp8m7ytt7z4 nginx.8 nginx: latest ip-172-31-26-234.ec2.internal 
Running Running 24 seconds ago 
zegs90ro15nn nginx.9 nginx:latest ip-172-31-37-135.ec2.internal 
Running Running 41 seconds ago 
qfkpvy28g1g6 nginx.10 nginx:latest ip-172-31-26-234.ec2.internal 
Running Running 24 seconds ago 


service Tasks Replacement on a Node Leaving the Swarm 


The desired state reconciliation in Docker Swarm mode ensures that the desired number of replicas are 
running if resources are available. If a node is made to leave a Swarm, the replicas running on the node are 
scheduled on another node. Starting with a mysql service replica running on each node in a three-node 
Swarm, you can make one worker node leave the Swarm. 


" docker swarm 1eave 
Node left the swarm. 


A replacement service task for the service task running on the shutdown node gets scheduled on 
another node. 


" s docker service ps mysql 

NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 

6zu7a59ejdxip3y90eu548hv5 mysql.l mysql ip-10-0-0-46.ec2.internal Running Running 3 minutes ago 
441cuufa7sa9móeatqbiq7vi3 mysql.2 mysql ip-10-0-0-28.ec2.internal Running Running about a minute ago 
blcdm8Bh6v86gl..pwp6zx3janv mysql.3 mysql ip-10-0-0-28.ec2.internal Running Running 4 seconds ago 
Or3oki4acf3d6ils5iazmg425 N mysql.3 mysql ip-10-0-0-106.ec2.internal Shutdown Running about a minute ago 


Make the other worker node also leave the Swarm. The service replicas on the other worker node also 
get shut down and scheduled on the only remaining node in the Swarm. 


" s docker service ps mysql 

NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 

62u7a59ejdxip3y90eu548hv5 mysql.1 mysql ip-10-0-0-46.ec2. internal Running Running 5 minutes ago 
dbdaxvl6lohlxrsxh5aobjxi8 mysq.2 mysql ip-10-0-0-46.ec2.internal Running Running 7 seconds ago 
44tcuufa7sagmbeatqbiq7vi3 N mysql.2 mysql ip-10-0-0-28.ec2.internal Shutdown Running 2 minutes ago 
216iu28xhshztm3bgtvy7ttk8 mysql.3 mysql ip-10-0-0-46.ec2.internal Running Running 7 seconds ago 
blcdm88h6v86glpwp6zx3janv \_ mysql.3 mysql ip-10-0-0-28.ec2.internal Shutdown Running about a minute ago 


Or3oki4acf3d6ils5iazmg425 N mysql.3 mysql ip-10-0-0-106.ec2.internal Shutdown Running 2 minutes ago 
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If only the replicas with desired state as running are listed, all replicas are listed as running on the 
manager node. 


"s docker service ps -f desired-state-running mysql 

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR 

62u7a59ejdxip3y90eu548hv5 mysql.1 mysql ip-10-0-0-46.ec2.internal Running Running 7 minutes ago 
dbdaxvlelohlxrsxh5aobjxi8 mysql.2 mysql ip-10-0-0-46.ec2.internal Running Running 2 minutes ago 


216iu28xhshztm3bgtvy7ttk8 mysql.3 mysql ip-10-0-0-46.ec2.internal Running Running 2 minutes ago 


Summary 


This chapter discussed service scaling in Swarm mode. Only a replicated service can be scaled and not a 
global service. A service may be scaled up to as many replicas as resources can support and can be scaled 
down to no replicas. Multiple services may be scaled using the same command. Desire state reconciliation 
ensures that the desired number of service replicas are running. The next chapter covers Docker service 
mounts. 
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Using Mounts 


A service task container in a Swarm has access to the filesystem inherited from its Docker image. The data is 
made integral to a Docker container via its Docker image. At times, a Docker container may need to store or 
access data on a persistent filesystem. While a container has a filesystem, it is removed once the container 
exits. In order to store data across container restarts, that data must be persisted somewhere outside the 
container. 


The Problem 


Data stored only within a container could result in the following issues: 
e The data is not persistent. The data is removed when a Docker container is stopped. 


e The data cannot be shared with other Docker containers or with the host filesystem. 


The Solution 


Modular design based on the Single Responsibility Principle (SRP) recommends that data be decoupled 
from the Docker container. Docker Swarm mode provides mounts for sharing data and making data 
persistent across a container startup and shutdown. Docker Swarm mode provides two types of mounts for 
services: 


e Volume mounts 
e Bind mounts 


The default is the volume mount. A mount for a service is created using the - -mount option of the 
docker service create command. 


Volume Mounts 


Volume mounts are named volumes on the host mounted into a service task's container. The named 
volumes on the host persist even after a container has been stopped and removed. The named volume may 
be created before creating the service in which the volume is to be used or the volume may be created at 
service deployment time. Named volumes created at deployment time are created just prior to starting a 
service task's container. If created at service deployment time, the named volume is given an auto-generated 
name if a volume name is not specified. An example of a volume mount is shown in Figure 6-1, in which 

a named volume mysql-scripts, which exists prior to creating a service, is mounted into service task 
containers at the directory path /etc/mysql/scripts. 
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Mount 
Of Type 
"volume" 


Service 
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/etc/mysql/scripts /etc/mysql/scripts 


Named Volume: Named Volume: Named Volume: 
mysgl-scripts mysgl-scripts mysql-scripts 





Node Node Node 


Figure 6-1. Volume mount 


Each container in the service has access to the same named volume on the host on which the container 
is running, but the host named volume could store the same or different data. 

When using volume mounts, contents are not replicated across the cluster. For example, if you put 
something into the mysql-scripts directory you're using, those new files will only be accessible to other 
tasks running on that same node. Replicas running on other nodes will not have access to those files. 


Bind Mounts 


Bind mounts are filesystem paths on the host on which the service task is to be scheduled. The host 
filesystem path is mounted into a service task's container at the specified directory path. The host filesystem 
path must exist on each host in the Swarm on which a task may be scheduled prior to a service being 
created. If certain nodes are to be excluded for service deployment, using node constraints, the bind mount 
host filesystem does not have to exist on those nodes. When using bind mounts, keep in mind that the 
service using a bind mount is not portable as such. If the service is to be deployed in production, the host 
directory path must exist on each host in the Swarm in the production cluster. 

The host filesystem path does not have to be the same as the destination directory path in a task 
container. As an example, the host path /db/mysql/data is mounted as a bind mount into a service's 
containers at directory path /etc/mysql/data in Figure 6-2. A bind mount is read-write by default, but 
could be made read-only at service deployment time. Each container in the service has access to the 
same directory path on the host on which the container is running, but the host directory path could store 
different or the same data. 
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Figure 6-2. Bind mount 


Swarm mode mounts provide shareable named volumes and filesystem paths on the host that persist 
across a service task startup and shutdown. A Docker image's filesystem is still at the root of the filesystem 
hierarchy and a mount can only be mounted on a directory path within the root filesystem. 

This chapter covers the following topics: 


e Setting the environment 

e Types of mounts 

e Creating anamed volume 

e Using a volume mount to get detailed info about a volume 
e Removing a volume 


e Creating and using a bind mount 


Setting the Environment 


Create a Docker for AWS-based Swarm consisting of one manager node and two worker nodes, as discussed 
in Chapter 3. The Docker for AWS Swarm will be used for one type of mount, the volume mount. For the bind 
mount, create a three-node Swarm consisting of one manager and two worker nodes on CoreOS instances. 
Creating a Swarm on CoreOS instances is discussed in Chapter 2. A CoreOS-based Swarm is used because 
Docker for AWS Swarm does not support bind mounts out-of-the-box. Obtain the public IP address of the 
manager instance for the Docker for AWS Swarm from the EC2 console, as shown in Figure 6-3. 
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Instance state — running D IPva Public IP. — 52:51. 115.180 
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Avallabdity zone — us-east-Tb Private IPs — 172.31.16.11 
Secunty groups — Docker-ManagervpcsG-SO2EXTKIWON , Secondary private IPs 


Docker-SwarmnWideSG-1TKYZOQS83JY 837 
view mnbound rules 


Figure 6-3. EC2 instances for Docker for AWS Swarm nodes 


SSH login into the manager instance. 


[root@localhost ~]# ssh -i "docker.pem" docker@52.91.115.180 
Welcome to Docker! 


List the nodes in the Swarm. A manager node and two worker nodes are listed. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
8yng7exfo5v74ymoe7hrsghxh | ip-172-31-33-230.ec2.internal Ready Active 
o0h7009a61ico7nit800e281g * ip-172-31-16-11.ec2.internal Ready Active Leader 


yzlv7c3qwcwozhxz439dbknj4 ^ ip-172-31-25-163.ec2.internal Ready Active 


Creating a Named Volume 


A named volume to be used in a service as a mount of type volume may either be created prior to creating 
the service or at deployment time. A new named volume is created with the following command syntax. 


docker volume create [OPTIONS] [VOLUME | 
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The options discussed in Table 6-1 are supported. 


Table 6-1. Options for the docker volume create Command for a Named Volume 


Option Description Type Default Value 
--driver, -d Specifies volume driver name string local 

- -label Sets metadata for a volume value [ | 

--name Specifies volume name string 

--opt, -o Sets driver specific options value map| | 


Create a named volume called hello using the docker volume create command. 


~ $ docker volume create --name hello 
hello 


Subsequently, list the volumes with the docker volume 1s command. The hello volume is listed in 
addition to other named volumes that may exist. 


~ $ docker volume ls 
DRIVER VOLUME NAME 
local hello 
You can find detailed info about the volume using the following command. 
docker volume inspect hello 


In addition to the volume name and driver, the mountpoint of the volume also is listed. 


~ $ docker volume inspect hello 


l 


{ 
"Driver": "local", 
"Labels": {}, 
"Mountpoint": "/var/lib/docker/volumes/hello/ data", 
"Name": "hello", 
"Options": {}, 
"Scope": "local" 
j 


The scope of a local driver volume is local. The other supported scope is global. A local volume is 
created on a single Docker host and a global volume is created on each Docker host in the cluster. 
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Using a Volume Mount 


Use the hello volume in the docker service create command with the - -mount option. The options 
discussed in Table 6-2 may be used both with bind mounts and volume mounts. 


Table 6-2. Options for Volume and Bind Mounts 


Option Required Description Default 


type NO Specifies the type of mount. One of three values volume 
may be specified: 


volume-Mounts is a named volume in a container. 
bind-Bind-mounts is a directory or file from the 
host into a container. 

tmpfs-Mounts is a tmpfs into a container. 


SIC Or Yes for type=bind only. The source directory or volume. The option has 
source No for type=volume different meanings for different types of mounts. 


type=volume: src specifies the name of the volume. 
If the named volume does not exist, it is created. 

If src is omitted, the named volume is created 

with an auto-generated name, which is unique on 
the host but may not be unique cluster-wide. An 
auto-generated named volume is removed when 
the container using the volume is removed. The 
docker service update command shuts down task 
containers and starts new task containers and so 
does scaling a service. volume source must not be 
an absolute path. 


type=bind: src specifies the absolute path to the 
directory or file to bind-mount. The directory path 
must be an absolute and not a relative path. The src 
option is required for a mount of type bind and an 
error is generated if it’s not specified. 

type=tmpfs: is not supported. 


dst or Yes Specifies the mount path inside a container. If 
destination the path does not exist in a container’s filesystem, 
or target the Docker engine creates the mount path before 


mounting the bind or volume mount. The volume 
target must be a relative path. 


readonly No A boolean (true/false) or (1/0) to indicate whether 
or ro the Docker Engine should mount volumes and bind 
read-write or read-only. If the option is not specified, 
the engine mounts the bind or volume read-write. 
If the option is specified with a value of true or 1 
or no value, the engine mounts the volume or bind 
read-only. If the option is specified with a value of 
false or 0, the engine mounts the volume or bind 
read-write. 
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Some of the mount options are only supported for volume mounts and are discussed in Table 6-3. 


Table 6-3. Options for Volume Mounts 
Option Required Description Default Value 


volume-driver NO Specifies the name of the volume-driver plugin to use ^ local 
for the volume. If a named volume is not specified 
in src, the volume-driver is used to create a named 
volume. 


volume-label NO Specifies one or more comma-separated metadata 
labels to apply to the volume. Example: volume- 
label=label-1=hello-world, label-2=hello. 


volume-nocopy No Applies to an empty volume that is mounted in a true or 1 
container at a mount path at which files and directories 
already existed. Specifies whether a container's 
filesystem files and directories at the mount path (dst) 
are to be copied to the volume. A host is able to access 
the files and directories copied from the container to 
the named volume. A value of true or 1 disables 
copying of files from the container's filesystem to the 
host volume. A value of false or 0 enables copying. 


volume-opt NO Specifies the options to be supplied to the volume- 
driver in creating a named volume if one does not 
exist. The volume-opt options are specified as a 
comma-separated list of key/value pairs. Example: 
volume-opt-1-option-1-value1,option-2-value2. 


A named volume has to exist on each host on which 
a mount of type volume is to be mounted. Creating a 
named volume on the Swarm manager does not also 
create the named volume on the worker nodes. The 
volume-driver and volume-opt options are used to 
create the named volume on the worker nodes. 


The options discussed in Table 6-4 are supported only with a mount of type tmpfs. 


Table 6-4. Options for the tmpfs Mount 


Option Required Description Default Value 
tmpfs-size | No Size ofthe tmpfs mount in bytes Unlimited value on Linux 
tmpfs-mode No Specifies the file mode ofthe tmpfsinoctal 1777 in Linux 


Next, we will use the named volume hello in a service created with Docker image tutum/hello-world. 
In the following docker service create command, the - -mount option specifies the src as hello and 
includes some volume- label labels for the volume. 
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~ $ docker service create \ 
--name hello-world \ 
--mount src-hello,dst-/hello,volume-label-"msg-hello",volume-label-"msg2-world" \ 
--publish 8080:80 \ 
--replicas 2 \ 
tutum/hello-world 


The service is created and the service ID is output. 


~ $ docker service create \ 
--name hello-world \ 
--mount src-hello,dst-/hello,volume-label-"msg-hello",volume-label-"msg2-world" \ 
--publish 8080:80 \ 
--replicas 2 \ 
tutum/hello-world 
1ly37072wyxkyw2jt60kdqoz 


OO v wv wv wv MW 


Two service replicas are created. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
8ily37072wyx hello-world replicated 2/2 tutum/hello-world:latest *:8080->80/tcp 
~ $ docker service ps hello-world 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

uw6coztxwqhf hello-world.1 tutum/hello-world:latest ip-172-31-25-163.ec2.internal 
Running Running 20 seconds ago 

cfkwefwadkki  hello-world.2 tutum/hello-world:latest ip-172-31-16-11.ec2.internal 
Running Running 21 seconds ago 


The named volume is mounted in each task container in the service. 
The service definition lists the mounts, including the mount labels. 


~ $ docker service inspect hello-world 


"Spec": { 
"ContainerSpec": { 
"Image": “tutum/hello-world: latest@sha256:0d57def8055178aafb4c7669cbc25e 
c17f0acdab97cc587F30150802da8F8d85", 
"Mounts": [ 
{ 
"Type": "volume", 
"Source": "hello", 
"Target": "/hello", 
"VolumeOptions": { 
"Labels": ( 
"msg": "hello", 
"msg2": "world" 


J, 
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In the preceding example, a named volume is created before using the volume in a volume mount. As 
another example, create a named volume at deployment time. In the following docker service create 
command, the - -mount option is set to type-volume with the source set to nginx-root. The named volume 
nginx-root does not exist prior to creating the service. 


~ $ docker service create \ 


>  --name nginx-service \ 
>  --replicas 3 \ 
> --mount type=volume, source="nginx-root” ,destination="/var/lib/nginx” , volume- 


label="type=nginx root dir" \ 
>  nginx:alpine 
rtzildok405mro3uhdkihtlnk 


When the command is run, a service is created. Service description includes the volume mount in 
mounts. 


~ $ docker service inspect nginx-service 


l 
"Spec": { 
"Name": "nginx-service", 
"Mounts": [ 
{ 


"Type": "volume", 
"Source": "nginx-root", 
"Target": "/var/lib/nginx', 
"VolumeOptions": { 
"Labels": ( 
"type": "nginx root dir" 


3 


The named volume nginx-root was not created prior to creating the service and is therefore created 
before starting containers for service tasks. The named volume nginx-root is created only on nodes on 
which a task is scheduled. One service task is scheduled on each of the three nodes. 


~ $ docker service ps nginx-service 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

pfqinizqmgur nginx-service.1 nginx:alpine ip-172-31-33-230.ec2.internal 
Running Running 19 seconds ago 

mn8h3p40chgs nginx-service.2 nginx:alpine ip-172-31-25-163.ec2.internal 
Running Running 19 seconds ago 

k8n5zzlnn46s nginx-service.3 nginx:alpine ip-172-31-16-11.ec2.internal 
Running Running 18 seconds ago 
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As a taskis scheduled on the manager node, a named volume called nginx-root is created on the 
manager node, as listed in the output of the docker volume 1s command. 


~ $ docker volume ls 


DRIVER VOLUME NAME 
local hello 
local nginx-root 


Service tasks and task containers are started on each of the two worker nodes. A nginx-root named 
volume is created on each of the worker nodes. Listing the volumes on the worker nodes lists the nginx-root 
volume. 


[root@localhost ~]# ssh -i "docker.pem" docker@34.229.86.64 
Welcome to Docker! 
~ $ docker volume ls 


DRIVER VOLUME NAME 
local hello 
local nginx-root 


[root@localhost ~]# ssh -i "docker.pem" docker@52.91.200.241 
Welcome to Docker! 
~ $ docker volume ls 


DRIVER VOLUME NAME 
local hello 
local nginx-root 


A named volume was specified in src in the preceding example. The named volume may be omitted as 
in the following service definition. 


~ $ docker service create ^ 

>  --name nginx-service-2 \ 

>  --replicas 3 \ 

> | --mount type-volume,destination-/var/lib/nginx  ' 
>  nginx:alpine 

q8ordkmkwqrwiwhmaemvcypc3 


The service is created with a replica and is scheduled on each of the Swarm nodes. 


~ $ docker service ps nginx-service-2 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

kz8d8k6bxp7u nginx-service-2.1 nginx:alpine ip-172-31-25-163.ec2.internal 
Running Running 27 seconds ago 

wd65qsmqixpg nginx-service-2.2 nginx:alpine ip-172-31-16-11.ec2.internal 
Running Running 27 seconds ago 

mbnmzldtaaed nginx-service-2.3 nginx:alpine ip-172-31-33-230.ec2.internal 
Running Running 26 seconds ago 
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The service definition does not list a named volume. 


~ $ docker service inspect nginx-service-2 


"Spec": { 
"Name": "nginx-service-2", 
"ContainerSpec": ( 
"Mounts": [ 
{ 


"Type": "volume", 
"Target": "/var/lib/nginx" 


Named volumes with auto-generated names are created when a volume name is not specified explicitly. 
One auto-generated named volume with an auto-generated name is created on each node on which a 
service task is run. One of the named volumes listed on the manager node is an auto-generated named 
volume with an auto-generated name. 


~ $ docker volume ls 


DRIVER VOLUME NAME 

local 305f1fa3673e811b3b320fad0e2dd5786567bcec49b3e66480eab2309101e233 
local hello 

local nginx-root 


As another example of using named volumes as mounts in a service, create a named volume called 
mysql-scripts fora MySQL database service. 


~ $ docker volume create --name mysql-scripts 
mysql-scripts 


The named volume is created and listed. 


~ $ docker volume ls 


DRIVER VOLUME NAME 

local 305f1fa3673e811b3b320fad0e2dd5786567bcec49b3e66480eab2309101e233 
local hello 

local mysql-scripts 

local nginx-root 
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The volume description lists the scope as local and lists the mountpoint. 


~ $ docker volume inspect mysql-scripts 


[ 
{ 
"Driver": "local", 
"Labels": {}, 
"Mountpoint": "/var/lib/docker/volumes/mysql-scripts/ data", 
"Name": "mysql-scripts", 
"Options": {}, 
"Scope": "local" 
j 
] 


Next, create a service that uses the named volume in a volume mount. 


~ $ docker service create \ 
>  --env MYSQL ROOT PASSWORD-'mysql'^ 
> | --mount type-volume,src-"mysql-scripts",dst- "/etc/mysgl/scripts", 
el-"msg-mysql",volume-label-"msg2-scripts" ^ 
--publish 3306:3306* 
--replicas 2 \ 
--name mysql ^ 
mysql 
cghaz4zoxurpyqil5ikngf4c1 


VV VM 


The service is created and listed. 


~ ¢ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
8ily37072wyx hello-world replicated 2/2 tutum/hello-world:latest *:8080->80/tcp 
cghaz4zoxurp ysql replicated 1/2 mysql: latest *:3306-»3306/tcp 


Listing the service tasks indicates that the tasks are scheduled on the manager node and one of the 
worker nodes. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
y59yhzwch2fj mysql.1 mysql:latest ip-172-31-33-230.ec2.internal 
Running Preparing 12 seconds ago 

zg7wrludkr84 mysql.2 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running less than a second ago 


The destination directory for the named volume is created in the Docker container. The Docker 
container on the manager node may be listed with docker ps and a bash shell on the container may be 
started with the docker exec -it «containerid» bash command. 
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~ $ docker ps 


CONTAINER ID IMAGE COMMAND 

CREATED STATUS PORTS NAMES 

a855826cdc75 mysql: latest "docker-entrypoint..." 

22 seconds ago Up 21 seconds 3306/tcp mysql.2.zg7wrludkr84zf 
8vhdkf8wnlh 


" $ docker exec -it a855826cdc75 bash 
root@a855826cd75:/# 


Change the directory to /etc/mysql/scripts in the container. Initially, the directory is empty. 


root@a855826cdc75:/# cd /etc/mysql/scripts 
root@a855826cdc75:/etc/mysql/scripts# ls -1l 
total O 
root@a855826cdc75:/etc/mysql/scripts# exit 
exit 


A task container for the service is created on one of the worker nodes and may be listed on the worker 
node. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND 

CREATED STATUS PORTS NAMES 

eb8d59cc2dff mysql:latest "docker-entrypoint..." 

8 minutes ago Up 8 minutes 3306/tcp mysql.1.xjmx7qviihyq2so7nOoxiimuq 


Start a bash shell for the Docker container on the worker node. The /etc/mysql/scripts directory on 
which the named volume is mounted is created in the Docker container. 


~ $ docker exec -it eb8d59cc2dff bash 
root@eb8d59cc2dff:/# cd /etc/mysql/scripts 
rootQeb8d59cc2dff:/etc/mysql/scriptsit exit 
exit 


If a service using an auto-generated named volume is scaled to run a task on nodes on which a task was 
not running previously, named volumes are auto-generated on those nodes also. As an example of finding 
the effect of scaling a service when using an auto-generated named volume as a mount in the service, create 
a MySQL database service with a volume mount. The volume mysql-scripts does not exist prior to creating 
the service; remove the mysql-scripts volume if it exists. 


~ $ docker service create \ 


>  --env MYSQL ROOT PASSWORD-'mysql'^ 

>  --replicas 1 \ 

> | --mount type-volume,src-"mysql-scripts",dst- "/etc/mysgl/scripts"N 
>  --name mysql ^ 

» mysql 


088ddf5pt4yb3yvr5s7elyhpn 
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The service task is scheduled on a node. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

xlix91njbaqo mysql.1 mysql:latest ip-172-31-13-122.ec2.internal 
Running Preparing 12 seconds ago 


List the nodes; the node on which the service task is scheduled is the manager node. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
o5hyue3hzuds8vtyughswbosl ip-172-31-11-41.ec2.internal Ready Active 
p6uuzp8pmoahcwexr3wdulxv ip-172-31-23-247.ec2.internal Ready Active 
qnk35mo1411x8jljp87ggnsnq * ip-172-31-13-122.ec2.internal Ready Active Leader 


A named volume mysql-scripts and an ancillary named volume with an auto-generated name are 
created on the manager node on which a task is scheduled. 


~ $ docker volume ls 


DRIVER VOLUME NAME 
local a2bc631f1b1da354d30aaea37935c65f9d99c5f0844d92341c6506f1e2aab1d55 
local mysql-scripts 


The worker nodes do not list the nysql1- scripts named volume, as a task is not scheduled on the 
worker nodes. 


~ $ docker volume ls 
DRIVER VOLUME NAME 


Scale the service to three replicas. A replica is scheduled on each of the three nodes. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
xlix91njbaqO mysql.1 mysql:latest ip-172-31-13-122.ec2.internal 
Running Running about a minute ago 

ifk7xuvfp9p2 mysql.2 mysql:latest ip-172-31-23-247.ec2.internal 
Running Running less than a second ago 

3c53fxgcjqyt mysql.3 mysql:latest ip-172-31-11-41.ec2.internal 
Running Running less than a second ago 
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A named volume mysql-scripts and an ancillary named volume with an auto-generated name are 
created on the worker nodes because a replica is scheduled. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.165.69.9 
Welcome to Docker! 


~ $ docker volume ls 


DRIVER VOLUME NAME 
local 431a792646d0b04b5ace49a32e6c0631eCc5e92F3dda57008b1987e4fe2a1b561 
local mysql-scripts 


[root@localhost ~]# ssh -i "docker.pem" docker@34.232.95.243 
Welcome to Docker! 


~ $ docker volume ls 


DRIVER VOLUME NAME 
local afb2401a9a916a365304b8aa0cc96b1be0c16146240375745c9829f2b61f180873 
local mysql-scripts 


The auto-generated named volumes are persistent and do not get removed when a service replica is 
shut down. The named volumes with auto-generated names are not persistent volumes. As an example, 
scale the service back to one replica. Two of the replicas shut down, including the replica on the manager 
node. 


~ $ docker service scale mysql=1 
mysql scaled to 1 
~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

3c53fxgcjayt mysql.3 mysql:latest ip-172-31-11-41.ec2.internal 
Running Running 2 minutes ago 


But the named volume mysql-scripts on the manager node is not removed even though no Docker 
container using the volume is running. 


~ $ docker volume ls 
DRIVER VOLUME NAME 
local mysql-scripts 


Similarly, the named volume on a worker node on which a service replica is shut down also does not get 


removed even though no Docker container using the named volume is running. The named volume with the 
auto-generated name is removed when no container is using it, but the mysql-scripts named volume does not. 
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Remove the volume mysql-scripts still does not get removed. 


~ $ docker service rm mysql 


mysql 

~ $ docker volume ls 

DRIVER VOLUME NAME 
local mysql-scripts 


Removing a Volume 


A named volume may be removed using the following command. 
docker volume rm «VOL» 
As an example, remove the named volume mysql-scripts. 


~ $ docker volume rm mysql-scripts 
mysql-scripts 


If the volume you try to delete is used in a Docker container, an error is generated instead and the 
volume will not be removed. Even a named volume with an auto-generated name cannot be removed if it's 
being used in a container. 


Creating and Using a Bind Mount 


In this section, we create a mount of type bind. Bind mounts are suitable if data in directories that already 
exist on the host needs to be accessed from within Docker containers. type-bind must be specified with the 
- -mount option when creating a service with mount of type bind. The host source directory and the volume 
target must both be absolute paths. The host source directory must exist prior to creating a service. The 
target directory within each Docker container of the service is created automatically. Create a directory on 
the manager node and then add a file called createtable.sql to the directory. 


coreQip-10-0-0-143 ~ $ sudo mkdir -p /etc/mysql/scripts 


coreQip-10-0-0-143 ~ $ cd /etc/mysql/scripts 
core@ip-10-0-0-143 /etc/mysql/scripts $ sudo vi createtable.sql 
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Save a SQL script in the sample SQL file, as shown in Figure 6-4. 


root@localhost:~ 





File Edit View Search Terminal Help 


CREATE TABLE wlslog(time stamp VARCHAR(255) PRIMARY KEY,category VARCHAR(255),ty\<) 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2| 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2| 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2/: 
INSERT INTO wlslog(time stamp, category, type,servername,code,msg) VALUES( 'Apr-8-2 


~ 
~ 


~ 


:wa 


Figure 6-4. Adding a SQL script to the host directory 





Similarly, create a directory and add a SQL script to the worker nodes. 
Create a service with a bind mount that’s using the host directory. The destination directory is specified 
as /scripts. 


core@ip-10-0-0-143 ~ $ docker service create \ 

> . --env MYSQL ROOT PASSWORD-'mysql' V 

> --replicas 3 \ 

> --mount type-bind,src-"/etc/mysql/scripts",dst-"/scripts" \ 
> --name mysql \ 

> mysql 

Okvk2hk2gigqyeem8x1r8qkvk 


Start a bash shell for the service container from the node on which a task is scheduled. The destination 
directory /scripts is listed. 


coreQip-10-0-0-143 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 
STATUS PORTS NAMES 

e71275e6c65c mysql:latest "docker-entrypoint.sh" 5 seconds ago 
Up 4 seconds 3306/tcp mysql.1.btqfrx7uffym2xvc441pubaza 


core@ip-10-0-0-143 ~ $ docker exec -it e71275e6c65c bash 
root@e71275e6c65c:/# ls -1 
drwxr-xr-x. 2 root root 4096 Jul 24 20:44 scripts 


Change the directory (cd) to the destination mount path /scripts. The createtable.sql script is listed 
in the destination mount path of the bind mount. 


root@e71275e6c65c:/# cd /scripts 
root@e71275e6c65c:/scripts# ls -1l 
-IW-I--r--. 1 root root 1478 Jul 24 20:44 createtable.sql 
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Each service task Docker container has its own copy of the file on the host. Because, by default, the 
mount is read-write, the files in the mount path may be modified or removed. As an example, remove the 
createtable.sql script from a container. 


core@ip-10-0-0-137 ^" $ docker exec -it 995b9455aff2 bash 
root@995b9455aff2:/# cd /scripts 
root@995b9455aff2:/scripts# ls -1 

total 8 

-Irw-r--r--. 1 root root 1478 Jul 24 20:45 createtable.sql 
root@995b9455aff2:/scripts# rm createtable.sql 
root@995b9455aff2:/scripts# ls -1 

total 0 

root@995b9455aff2:/scripts# 


A mount may be made read-only by including an additional option in the --mount arg, as discussed 
earlier. To demonstrate a readonly mount, first remove the mysql service that’s already running. Create 
a service and mount a readonly bind with the same command as before, except include an additional 
readonly option. 


core@ip-10-0-0-143 ~ $ docker service create \ 


> . --env MYSQL ROOT PASSWORD-'mysql' ^ 

> --replicas 3 \ 

> --mount type-bind,src-"/etc/mysql/scripts",dst-"/scripts",readonly \ 
> --name mysql \ 

> mysql 


c27se8vfygk2z57rtswentrix 


A bind of type mount which is readonly is mounted. 
Access the container on a node on which a task is scheduled and list the sample script from the host 
directory. 


coreQip-10-0-0-143 ~ $ docker exec -it 3bf9cf777d25 bash 
root@3bf9cf777d25:/# cd /scripts 
root@3bf9cf777d25:/scripts# ls -1 

-IWw-r--r--. 1 root root 1478 Jul 24 20:44 createtable.sql 


Remove, or try to remove, the sample script. An error is generated. 


root@3bf9cf777d25:/scripts# rm createtable.sql 
rm: cannot remove 'createtable.sql': Read-only file system 


Summary 


This chapter introduced mounts in Swarm mode. Two types of mounts are supported— bind mount and 
volume mount. A bind mount mounts a pre-existing directory or file from the host into each container of a 
service. A volume mount mounts a named volume, which may or may not exist prior to creating a service, 
into each container in a service. The next chapter discusses configuring resources. 
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CHAPTER 7 


Configuring Resources 


Docker containers run in isolation on the underlying OS kernel and require resources to run. Docker Swarm 
mode supports two types of resources—CPU and memory—as illustrated in Figure 7-1. 


Docker Swarm Mode 





Figure 7-1. Types of resources supported by Docker Swarm mode 


The Problem 


By default, Docker Swarm mode does not impose any limit on how many resources (CPU cycles or memory) 
a service task may consume. Nor does Swarm mode guarantee minimum resources. Two issues can result if 
no resource configuration is specified in Docker Swarm mode. 

Some of the service tasks could consume a disproportionate amount of resources, while the other 
service tasks are not able to get scheduled due to lack of resources. As an example, consider a node 
with resource capacity of 3GB and 3 CPUs. Without any resource guarantees and limits, one service 
task container could consume most of the resources (2.8GB and 2.8 CPUs), while two other service task 
containers each have only 0.1GB and 0.1 CPU of resources remaining to be used and do not get scheduled, 
as illustrated in Figure 7-2. A Docker service task that does not have enough resources to get scheduled is 
put in Pending state. 
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2.8 GB 
2.8 CPUs 






3 CPUs 
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0.1 CPUs 






Docker 
Containers 





0.1 GB 
0.1 CPUs 








Figure 7-2. Unequal allocation of resources 


The second issue that can result is that the resource capacity of a node can get fully used up without any 
provision to schedule any more service tasks. As an example, a node with a resource capacity of 9GB and 9 
CPUs has three service task containers running, with each using 3GB and 3 CPUs, as illustrated in Figure 7-3. 
If a new service task is created for the same or another service, it does not have any available resources on 
the node. 






Node 
Capacity: 
9 GB, 
9 CPUs 






Docker 
Containers 


Figure 7-3. Fully resource-utilized node 


The Solution 


Docker Swarm mode has a provision to set resource guarantees (or reserves) and resource limits, as 
illustrated in Figure 7-4. A resource reserve is the minimum amount of a resource that is guaranteed or 
reserved for a service task. A resource limit is the maximum amount of a resource that a service task can use 
regardless of how much of a resource is available. 
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CPU & Memory 
Resources 


Resource Reserves Resource Limits 





Figure 7-4. Managing Swarm resources with resource reserves and limits 


With resource reserves, each service task container can be guaranteed 1 CPU and 1GB in the issue 
discussed previously, as illustrated in Figure 7-5. 






Node 
Capacity: 
3 GB, 
3 CPUs 






Docker 
Containers 


Figure 7-5. Resource allocation with resource reserves set 


And, if resource limits are implemented for service task containers, excess resources would be available 
to start new service task containers. In the example discussed previously, a limit of 2GB and 2 CPUs per 
service task would keep the excess resources of 3GB and 3 CPUs available for new service task containers, as 
illustrated in Figure 7-6. 
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Figure 7-6. Resource allocation with resource limits set 


This chapter covers the following topics: 
e Setting the environment 
e Creating a service without resource specification 
e Reserving resources 
e Setting resource limits 
e Creating a service with resource specification 
e Scaling and resources 
e Reserved resources must be less than resource limits 
e Rolling update to set resource limits and reserves 


e Resource usage and node capacity 


Setting the Environment 


Create a three-node Swarm on Docker for AWS with one manager node and two worker nodes. Creating a 
Swarm on Docker for AWS is discussed in Chapter 3. We use the three-node Swarm created in Chapter 6 for 
this chapter also. Obtain the public IP address of the Swarm manager instance, as shown in Figure 7-7. 
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Launch Instance Connect Actions ¥ 
CL oto 


QO Filter by tags and attributes or search by keyword Q 11t030f3 
Name * Instance ID * Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 
Docker-worker i-083e010a39e00271 Q micro us-east-1b @ running Q 2/2 checks None Ye e¢2-52-91-200-241.co 
@ Docker-Manager i-04812ba54e249c99c — 2. micro us-east-1b @ running © 2/2 checks None Me e¢2-52-91-115-180.co 
Docker-worker i-0a0d7ba92b62454 t2.micro us-east-1¢ @ running © 2/2 checks None We. e¢2-34-229-86-64.com 


instance: | i-04812b254e249c99c (Docker-Manager) Public ONS: ec2-52-91-115-180.compute-1.amazonaws.com SFr - 
Description Status Checks Monitoring Tags 
Instance ID i-04812ba54e249¢9%e Pubic DNS (IPv4)  ec2-52-91-115-180.compute- 


nstance state — running \  IPv4 Public IP 5291.115.180 
Instance type — t2.micro IPv6 IPs 
Elastic IPs Private ONS — 1p-172-31-16-11 ec2 intema 
Availability zone — us-east-1b Private IPs 172 31.16 11 
Security groups — Docker-ManagerVpc SG-SO2EXTKIWOI2 Secondary private IPs 


Docker-SwarmWideSG- 17KYZQ83JY9GZ 
view inbound rules 


Figure 7-7. EC2 instances for Swarm nodes 


SSH login into the manager instance with user as “docker”. 


[root@localhost ~]# ssh -i "docker.pem" docker@52.91.115.180 
Welcome to Docker! 


List the Swarm nodes; a manager node and two worker nodes are listed. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
8ynq7exfo5v74ymoe7hrsghxh ^ ip-172-31-33-230.ec2.internal Ready Active 
o0h7009a61ico7nit800e281g * ip-172-31-16-11.ec2.internal Ready Active Leader 


yzlv7c3qucwozhxz439dbknj4 | ip-172-31-25-163.ec2.internal Ready Active 


Creating a Service Without Resource Specification 


We start by creating a service without any resource specification. Create a MySQL database service without 
setting any resource reserves or limits. 


docker service create \ 
--env MYSQL ROOT PASSWORD- 'mysql'N 
--replicas 1 \ 
--name mysql \ 

mysql 


A single service replica is created. The output of the command is the service ID (shown in italics). 


~ $ docker service create \ 


>  --env MYSQL ROOT PASSWORD-'mysql'^ 
>  --replicas 1 \ 

>  --name mysql ^ 

» mysql 


2kcq6cf72t4wu94000k3sax41 
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List the services; the mysql service is listed. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
2kcq6cf72t4w mysql replicated 1/1 mysql: latest 


List the service tasks. The only service task is running on a worker node. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

sccqv4k9r22h mysql.1 mysql: latest ip-172-31-33-230.ec2.internal 
Running Running 10 seconds ago 


On inspecting the service, the container spec does not include any resources, limits, or reserves. The 
single service task may use all of the available resources on the node on which it’s scheduled. 


~ $ docker service inspect mysql 


"Resources": { 
"Limits": {}, 
"Reservations": {} 


)» 


Reserving Resources 


Swarm mode provides two options for resource reserves in the docker service create and docker 
service update commands, as listed in Table 7-1. 


Table 7-1. Options for Resource Reserves 


Option Description Default Value 
--reserve-cpu Reserve CPUs. A value of 0.000 implies no reserves are set. 0.000 
--reserve-memory Reserve memory. A value of 0 implies no reserves are set. 0 


setting Resource Limits 


Swarm mode provides two options for resource limits in the docker service createand docker service 
update commands, as discussed in Table 7-2. 


Table 7-2. Options for Resource Limits 


Option Description Default Value 
--limit-cpu Limit CPUs 0.000 
--limit-memory Limit Memory 0 
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Creating a Service with Resource Specification 


Next, create a service using resource specification. Set resource reserves of 0.25 CPUs and 128MB and 
resource limits of 1 CPU and 256MB. Remove the mysql service previously created before creating a new 
service with resources defined. The output of the command is the service ID (shown in italics). 


~ $ docker service rm mysql 
mysql 
~ $ docker service create ^ 
>  --env MYSQL ROOT PASSWORD='mysql' \ 
--replicas 1 \ 
--name mysql ^ 
--reserve-cpu .25 --limit-cpu 1 --reserve-memory 128mb --limit-memory 256mb ^ 
mysql 
abwq9budo7 joydo00u32z2b047 


> 
> 
> 
> 


On inspecting the service, the resources limits and reserves are listed, which contrasts with the empty 
settings for resources when a service is created without the resources definition. 


~ $ docker service inspect mysql 
"Resources": { 

"Limits": { 
"NanoCPUs": 1000000000, 
"MemoryBytes": 268435456 

32 

"Reservations": { 
"NanoCPUs": 250000000, 
"MemoryBytes": 134217728 


J, 


scaling and Resources 


Before scaling up a service, it may be suitable to determine the node capacity in terms of CPU and memory 
resources. As all three nodes in the Swarm are identical, the node capacity on one node is the same as on 
the other nodes. The node capacity is 1 CPU and 1GB, as listed in the output of the docker node inspect 
command. 


~ $ docker node inspect ip-172-31-16-11.ec2.internal 


l 


"Resources": { 
"NanoCPUs": 1000000000, 
"MemoryBytes": 1039040512 


3) 
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The CPU limit on each service task created in the preceding section is also 1 CPU. When scaling, the 
total of the resource limits for all service tasks on a node may exceed the node's capacity. However, the total 
of resource reserves must not exceed node capacity. 

As an example, scale to five replicas. 


~ $ docker service scale mysql=5 
mysql scaled to 5 


Scaling to five schedules two replicas on the manager node, two replicas on one of the worker nodes, 
and one replica on the other worker node. The aggregate of the resource limits on the worker nodes is 
exceeded but the aggregate of resource reserves are within the node's capacity. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

npc5r7xf98fg mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 2 minutes ago 

xokdhowntpOw mysql.2 mysql:latest ip-172-31-25-163.ec2.internal 
Running Running 13 seconds ago 

b6h4bsf7xzdc mysql.3 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 12 seconds ago 

j1d7ti7nb80u mysql.4 mysql:latest ip-172-31-33-230.ec2.internal 
Running Running 13 seconds ago 

w6to9pxcdbm5 mysql.5 mysql:latest ip-172-31-25-163.ec2.internal 
Running Running 13 seconds ago 


Reserved Resources Must Not Be More Than Resource Limits 


The resource limits are not taken into consideration when scheduling a service task, only the resource 
reserves are. Not setting the reserves (whether limits are set or not and whether limits exceed node capacity) 
schedules the service task if the resources required to run a task are within the node capacity. Resource 
reserves must not exceed resource limits or a service task may not get scheduled or might fail after a while. 
As an example, delete the mysql service and create a new service where the resource reserves exceed 
resource limits. The output of the command is the service ID (shown in italics). 


~ $ docker service rm mysql 
mysql 
~ $ docker service create ^ 
>  --env MYSQL ROOT PASSWORD='mysql'\ 
--replicas 1 \ 
--name mysql \ 
--reserve-cpu .75 --limit-cpu .5 --reserve-memory 256mb --limit-memory 128mb \ 
mysql 
srot5vr8x7v7iml2awc3fxbiu 


> 
> 
> 
> 


The service is created and even scheduled. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

pmcjrj6p3wfp mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 20 seconds ago 
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The service configuration has the resource reserves exceeding the resource limits. 


~ $ docker service inspect mysql 


) 
"Resources": ( 
"Limits": ( 
"NanoCPUs": 500000000, 
"MemoryBytes": 134217728 
32 
"Reservations": { 
"NanoCPUs": 750000000, 
"MemoryBytes": 268435456 
} 
) 


The resource reserves are within the node capacity, but because the resource limits are less than the 
resource reserves, the newly started service task fails and is shut down. The service task keeps getting 
restarted and shut down. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
vjcnjkwfdfkb mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 16 seconds ago 

pxdku8pxviyn V mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Shutdown Failed 21 seconds ago "task: non-zero exit (1)" 
pmcjrj6p3wfp V mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Shutdown Failed about a minute ago "task: non-zero exit (1)" 


The service task resource limits can be the same as the resource reserves. Remove the mysql service and 
create it again with the resource limits the same as the resource reserves. The output of the command is the 
service ID (shown in italics). 


~ $ docker service rm mysql 
mysql 
~ $ docker service create \ 


> --env MYSQL ROOT PASSWORD-'mysql'N 

>  --replicas 1 \ 

>  --name mysql ^ 

>  --reserve-cpu .5 --limit-cpu .5 --reserve-memory 256mb --limit-memory 256mb ^ 
» mysql 

81bu63v97p9rm81xfyxv9k11e 


The service is created and the single task is scheduled. The service task does not fail as when the 
resource reserves exceeded the resource limit. 
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~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

4i1fpha53abs mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 33 seconds ago 


And a Docker container is started. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED STATUS 
PORTS NAMES 
14d5553f0393 mysql: latest "docker-entrypoint..." 34 seconds ago Up 33 seconds 


3306/tcp mysql.1.411fpha53abs l4qky9dgafo8t 


Rolling Update to Modify Resource Limits and Reserves 


This section demonstrates a rolling update to set new CPU and memory limits and reserves. The service 
created in the previous section is used for updating in this section. Using the docker service update 
command, update the CPU and memory reserves and limits. The output of the command is the service 
name mysql (shown in italics). 


~ $ docker service update --reserve-cpu 1 --limit-cpu 2 --reserve-memory  256mb 
--limit-memory 512mb mysql 
mysql 


The resources are updated. Updating the resource specification for a service shuts down the service 
replica and starts a new replica with the new resource specification. 


~ ¢ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
81bu63v97p9r mysql replicated 1/1 mysql:latest 

~ $ docker service ps mysql 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

xkis4mirgbtv mysql.1 mysql:latest ip-172-31-33-230.ec2.internal 
Running Running 14 seconds ago 

4i1fpha53abs V mysql.1 mysql: latest ip-172-31-16-11.ec2. internal 
Shutdown Shutdown 15 seconds ago 


The service resources configuration is updated. 


~ $ docker service inspect mysql 


[ 
Jo 
"Resources": { 
"Limits": { 
"NanoCPUs": 2000000000, 
"MemoryBytes": 536870912 
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"Reservations": ( 
"NanoCPUs": 1000000000, 
"MemoryBytes": 268435456 


Jy 


Resource Usage and Node Capacity 


Resource usage cannot exceed node capacity. On the three-node Swarm (one manager and two worker 
nodes), recall that the node capacity is 1GB and 1 CPU. 

Remove the mysql service that's already running and create a mysql service with three replicas that requests 
4GB of memory. The service is created. The output of the command is the service ID (shown in italics). 


~ $ docker service rm mysql 

mysql 

~ $ docker service create ^ 

>  --env MYSOL ROOT PASSWORD- 'mysql'N 
--replicas 3 \ 
--name mysql \ 
--reserve-memory=4GB\ 

mysql 

cgrihwij2znn4jkfe6hswxgr7 


> 
> 
> 
> 


None of the service replicas is scheduled, as indicated by the Replicas column value of 0/3, because 
the requested capacity is more than the node capacity of a single node. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
cgrihwij2znn mysql replicated 0/3 mysql:latest 


The Current State ofthe replicas is listed as Pending. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 
DESIRED STATE CURRENT STATE ERROR PORTS 
vm7z20krx3j6 mysql.1 mysql:latest 

Running Pending 19 seconds ago 

exmsheo144ef mysql.2 mysql: latest 

Running Pending 19 seconds ago 

kiset9poqz2s mysql .3 mysql:latest 

Running Pending 19 seconds ago 


If a service that was previously running with all replicas is scaled up, some or all of the replicas could 
get de-scheduled. This happens if the resources required to run the new replicas exceed the available node 
capacity. As an example, remove the mysql service and create a new mysql service with resource settings 
within the provision of a node. The output of the command is the service ID (shown in italics). 
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~ $ docker service rm mysql 


~ ¢ docker service create \ 


>  --env MYSOL ROOT PASSWORD-'mysql'^ 

>  --replicas 1 \ 

>  --name mysql \ 

>  --reserve-cpu .5 --reserve-memory 512mb  \ 
» mysql 


ysef8nozmhuwa7sxerc9jwjqx 

The service is created and the single replica is running as indicated by the Replicas column value of 1/1. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 1/1 mysql:latest 


Incrementally scale up the service to determine if all of the service replicas are scheduled. First, scale up 
to three replicas. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


The service description lists 3/3 Replicas as running. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 3/3 mysql:latest 


The service replicas are scheduled, one replica on each node in the Swarm, using the spread scheduling 
strategy, which is discussed in more detail in Chapter 8. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

8kkkdnsol690 mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 51 seconds ago 

k209uge36bih mysql.2 mysql:latest ip-172-31-25-163.ec2.internal 
Running Running 16 seconds ago 

oiublpclz9eu mysql.3 mysql:latest ip-172-31-33-230.ec2.internal 
Running Running 16 seconds ago 


Scale the mysql service further up to replicas. 


~ $ docker service scale mysql-10 
mysql scaled to 10 


Only 3/10 of the replicas are listed as running. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 3/10 mysql:latest 
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Some of the replicas are Allocated but not scheduled for running on any node due to insufficient 
resources. The service replicas not running are listed with Current State set to Pending. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


8kkkdnsol690 
Running 
k209uge36bih 
Running 
oiublpclz9eu 
Running 
u807b7hoqvqc 
Running 
jh2ep10sonxy 
Running 
8d190sxa4fwf 
Running 
k8hba8j509vi 
Running 
ettk65bpin3b 
Running 
i3otbqfsfvr7 
Running 
sxdi97006d3b 
Running 


NAME 

CURRENT 
mysql.1 
Running 
mysql.2 
Running 
mysql.3 
Running 
mysql.4 
Pending 
mysql.5 
Pending 
mysql.6 
Pending 
mysql.7 
Pending 
mysql.8 
Pending 
mysql.9 
Pending 


mysql.10 


Pending 


IMAGE 


STATE ERROR 


mysql:latest 
about a minute ago 
mysql:latest 
35 seconds ago 
mysql:latest 
35 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 
mysql:latest 
7 seconds ago 


NODE 


PORTS 
ip-172-31-16-11.ec2.internal 


ip-172-31-25-163.ec2.internal 


ip-172-31-33-230.ec2.internal 


Adding one or more new worker nodes could make the service reconcile its desired state and cause all 
the replicas to run. To demonstrate next, we scale up the CloudFormation stack to increase the number of 


worker nodes. 


Scaling Up the Stack 


To scale up the CloudFormation stack, select the Docker stack in the CloudFormation > Stacks table and 
choose Actions > Update Stack, as shown in Figure 7-8. 


(d CloudFormation v 





Stacks 


è 


View/Eda temptate in Designer 


Figure 7-8. Choosing Actions > Update Stack 


Status Description 


The Update Docker Stack wizard starts. It’s similar to the Create Stack wizard. In the Select Template, 
click on Next without modifying any settings. In Specify Details, increase Number of Swarm Worker Nodes? 
to 10, as shown in Figure 7-9. Click on Next. 
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(p CloudFormation ~ Stacks > Stack Detail > Update Stack 


Update Docker stack 


Select Template Specify Details 
| Specity Details 
Options Specify parameter values. You can use or change the default parameter values which are defined in the AWS CloudFormation template Leam more 
Review 
Stack name Docker 
Parameters 
Swarm Size 


Number of Swarm 
managers? 


È Number of Swarm worker 
nodes? 


1 v Number of Swarm manager nodes (1, 3, 5) 


| 10 Number of worker nodes in the Swarm (0- 1000) 


Figure 7-9. Increasing the number of worker nodes to 10 


In Preview Your Changes, click on Update, as shown in Figure 7-10. 


Preview your changes 


Based on your input, CloudFormation will change the following resources. For more information, choose View change set details 


Action Logical ID Physical ID Resource type Replacement 
NodeAsg Docker-NodeAsg-1OUAOXESLISSJ AWS--AutoScaling--AutoScalingGroup False 


Cancel Previous D: 





Figure 7-10. Click Update to preview your changes 


When the update completes, the stack's status becomes UPDATE COMPLETE, as shown in Figure 7-11. 


(p CloudFormation v Stacks 


Eme 0e 


Filter. Ace ~ 
$t3ck Name Created Time 
+ Docker 2017-07-24 09 22-37 UTC-0700 


Figure 7-11. Stack update is complete 
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Showing 1 stack 
Status Description 
UPDATE COMPLETE Docker CE for AWS 17.06.0-ce (17 06 0-ce-aws2) 
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The Swarm gets eight new worker nodes, for a total of 10 worker nodes. List the service description 
periodically (after an interval of few seconds) and, as new worker nodes are created, new replicas start to 
reconcile the current state with the desired state. The number of replicas in the Replicas column increases 
gradually within a few seconds. All the replicas for the mysql service start running, as indicated by 10/10 in 
the service listing. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 3/10 mysql:latest 

~ $ docker service ls 

ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 6/10 mysql:latest 

~ $ docker service ls 

ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 9/10 mysql:latest 

~ $ docker service ls 

ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 10/10 mysql:latest 


Listing the service replicas lists all replicas as Running. The previously Pending replicas are scheduled 
on the new nodes. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

8kkkdnsolo90 mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 

Running Running 7 minutes ago 

k209uge36bih mysql.2 mysql:latest ip-172-31-25-163.ec2.internal 
Running Running 6 minutes ago 

oiublpclz9eu mysql.3 mysql:latest ip-172-31-33-230.ec2.internal 
Running Running 6 minutes ago 

u807b7hoqvqc mysql.4 mysql:latest ip-172-31-11-105.ec2.internal 
Running Running 45 seconds ago 

jh2ep10sonxy mysql.5 mysql:latest ip-172-31-13-141.ec2.internal 
Running Running about a minute ago 

8d190sxa4fwf mysql.6 mysql:latest ip-172-31-24-10.ec2.internal 

Running Running about a minute ago 

k8hba8j509vi mysql.7 mysql: latest ip-172-31-0-114.ec2. internal 

Running Running 55 seconds ago 

ettk65bpin3b mysql.8 mysql:latest ip-172-31-5-127.ec2.internal 

Running Running about a minute ago 

i3otbqfsfvr7 mysql.9 mysql:latest ip-172-31-35-209.ec2.internal 
Running Running 24 seconds ago 

sxdi97006d3b mysql.10 mysql:latest ip-172-31-21-57.ec2.internal 

Running Running 49 seconds ago 


If the stack is updated again to decrease the number of worker nodes, some of the replicas shut down 
and are de-scheduled. After decreasing the number of worker nodes, the Replicas column lists only 5/10 
replicas as running. 
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~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
ysef8no2mhuw mysql replicated 5/10 mysql:latest 


Some of the service tasks are listed as Shutdown because some of the worker nodes have been removed 
from the Swarm. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

8kkkdnsolo90 mysql.1 mysql:latest ip-172-31-16-11.ec2.internal 
Running Running 10 minutes ago 

ulknt3e5zxy1 mysql.2 mysql:latest 

Ready Pending 3 seconds ago 

k209uge36bih V mysql.2 mysql:latest ip-172-31-25-163.ec2.internal 
Shutdown Running 14 seconds ago 

oiublpclz9eu mysql.3 mysql:latest ip-172-31-33-230.ec2.internal 
Running Running 9 minutes ago 

mh2fpioi441k mysql.4 mysql:latest 

Running Pending 3 seconds ago 

u807b7hO0qvqc V mysql.4 mysql:latest v53huw84hskqsb3e800a2pmun 
Shutdown Running about a minute ago 

jzghd72nkozc mysql.5 mysql:latest 

Ready Pending 3 seconds ago 

jh2ep10sonxy V mysql.5 mysql:latest ip-172-31-13-141.ec2.internal 
Shutdown Running 14 seconds ago 

8d190sxa4fwf mysql.6 mysql:latest ip-172-31-24-10.ec2.internal 
Running Running 4 minutes ago 

dlcgstxxkd9t mysql.7 mysql:latest 

Running Pending 3 seconds ago 

ziqslz7u9d9l V mysql.7 mysql:latest ip-172-31-43-179.ec2.internal 
Shutdown Assigned 57 seconds ago 

k8hba8j509vi V mysql.7 mysql: latest opidzvmt5eyc74l6pcl5ut64p 
Shutdown Running about a minute ago 

ettk65bpin3b mysql .8 mysql: latest ip-172-31-5-127.ec2.internal 
Running Running 4 minutes ago 

i3otbqfsfvr7 mysql.9 mysql:latest ip-172-31-35-209.ec2.internal 
Running Running 3 minutes ago 

sxdi97006d3b mysql.10 mysql:latest ip-172-31-21-57.ec2.internal 
Running Running 12 seconds ago 


Summary 


This chapter discussed the resources model of Docker Swarm mode, which is based on resource reserves 
and resource limits. Reserved resources cannot be more than resource limits and resource allocation to 


service tasks is limited by the node capacity. The next chapter discusses scheduling in Docker Swarm mode. 
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CHAPTER 8 


Scheduling 


In Chapter 2, the Docker Swarm was introduced. In Chapter 4, Docker Swarm services were introduced. 

A service consists of zero or more service tasks (replicas), which it schedules on the nodes in a Swarm. 

The desired state of a service includes the number of tasks that must be run. Scheduling is defined as the 
process of placing a service task that is required to be run on a node in the Swarm to keep the desired state 
of a service, as illustrated in Figure 8-1. A service task may only be scheduled on a worker node. A manager 
node is also a worker node by default. 


scheduling d 


Figure 8-1. Scheduling 


The Problem 


Without a scheduling policy, the service tasks could get scheduled on a subset of nodes in a Swarm. As 
an example, all three tasks in a service could get scheduled on the same node in a Swarm, as illustrated in 
Figure 8-2. 
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Node 

Docker Swarm 
Service Node 
Node 


Figure 8-2. Avoid scheduling all tasks on one node 


Not using a scheduling policy could lead to the following problems: 


e  Underutilization of resources in a Swarm—]1f all the tasks are scheduled on a single 
node or a subset of nodes, the resource capacity of the other nodes is not utilized. 


e Unbalanced utilization of resources—If all the tasks are scheduled on a single node or 
a subset of nodes, the resources on the nodes on which the tasks are scheduled are 
over-utilized and the tasks could even use up all the resource capacity without any 
scope for scaling the replicas. 


e Lack of locality—Clients access a service's tasks based on node location. If all the 
service tasks are scheduled on a single node, the external clients that are accessing 
the service on other nodes cannot access the service locally, thereby incurring a 
network overhead in accessing a relatively remote task. 


e Single point of failure—If all services are running on one node and that node has a 
problem, it results in downtime. Increasing redundancy across nodes obviates that 
problem. 


The Solution 


To overcome the issues discussed in the preceding section, service task scheduling in a Docker Swarm is 
based on a built-in scheduling policy. Docker Swarm mode uses the spread scheduling strategy to rank nodes 
for placement of a service task (replica). Node ranking is computed for scheduling of each task and a task is 
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scheduled on the node with the highest computed ranking. The spread scheduling strategy computes node 
rank based on the node's available CPU, RAM, and the number of containers already running on the node. The 
spread strategy optimizes for the node with the least number of containers. Load sharing is the objective of the 
spread strategy and results in tasks (containers) spread thinly and evenly over several machines in the Swarm. 
The expected outcome of the spread strategy is that if a single node or a small subset of nodes go down or 
become available, only a few tasks are lost and a majority of tasks in the Swarm continue to be available. 


Note Because a container consumes resources during all states, including when it is exited, the spread 
strategy does not take into consideration the state of a container. It is recommended that a user remove 
stopped containers, because a node that would otherwise be eligible and suitable for scheduling a new task 
becomes unsuitable if it has several stopped containers. 


The spread scheduling strategy does not take into consideration for which service a task is scheduled. 
Only the available and requested resources are used to schedule a new task. Scheduling using the spread 
scheduling policy is illustrated in Figure 8-3. 





Node 

Docker Swarm 
Service Node 
Node 





Figure 8-3. Using the spread scheduling policy 
As a hypothetical example: 
1. Start with three nodes, each with a capacity of 3GB and 3 CPUs and no containers 


running. 
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Create a mysql service with one replica, which requests resources of 1GB and 
1 CPU. The first replica gets scheduled randomly on one of the three nodes in 
the Swarm as all nodes have the same ranking. If all the nodes have the same 
ranking, a new task gets scheduled randomly on one of the nodes. 


Scale the mysql service to three tasks. As one of the nodes is already loaded, 
the two new tasks are scheduled on the other two nodes, spreading one task to 
each node. 


Scale the mysql service to five tasks. Two new tasks must be started and all the 
nodes have the same ranking because they have the same available resource 
Capacity and the same number of containers running. The two new tasks are 
scheduled randomly on two of the nodes. As a result, two nodes have two tasks 
each and one node has one task. 


Create another service for the nginx server with a desired state of two tasks, 
with each task requesting 0.5GB and 0.5 CPU. Both the tasks are scheduled on 
the node that has only the task of the mysql service, as it is the least loaded. As 
a result, two nodes have two tasks of mysql service and an available capacity of 
1GB and 1 CPU, and one node has two tasks of nginx service and one task of 
mysql service and also an available resource capacity of 1GB and 1 CPU. 


Scale the nginx service to three. Even though all nodes have the same available 
CPU and RAM, the new task is not scheduled randomly on one of the three 
nodes, but is scheduled on the node with the least number of containers. As a 
result, the new nginx task gets scheduled randomly on one of the nodes, with 
two tasks of mysql each. If the nodes have the same available CPU and RAM, the 
node with fewer containers (running or stopped) is selected for scheduling the 
new task. 


This chapter covers the following topics: 


Setting the environment 

Creating and scheduling a service—the spread scheduling 
Desired state reconciliation 

Scheduling tasks limited by node resource capacity 

Adding service scheduling constraints 

Scheduling on a specific node 

Adding multiple scheduling constraints 

Adding node labels for scheduling 

Adding, updating, and removing service scheduling constraints 


Spread scheduling and global services 


Setting the Environment 


Create a CloudFormation stack using Docker for AWS consisting of one manager node and two worker 
nodes. Docker for AWS was introduced in Chapter 3. The stack is shown in Figure 8-4. 


(D CloudFormation ~ Stacks 
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o 
e Introducing StackSets 
AWS StackSet is a container for a set of AWS CloudFormation stacks and allows you to create stacks across multiple AWS Accounts and AWS Regions. Open Ine Stacksets console to gel 
started 
Desagn template | C uü 
Filter: Active ~ Showing 1 sack 
Stack Name Created Time Status Deseription 
Docker 2017-07-25 09 42:40 UTC-0700 CREATE COMPLETE Docker CE Tor AWS 17.08. 0-ce (17.06 O-ce-awsz) 


Figure 8-4. CloudFormation stack 


The three EC2 instances in the stack are shown in Figure 8-5. 





4 


Oo Filter by tags and attributes or search by keyword eo 
Name - Instance ID a Instance Type ~ AvailabilityZone ~ Instance State ~ Status Checks ~ Alarm Status 
Doc ker-worker i-Olagez4ocboeddebb — t2 miro us-east-1a @ running @ 2/2 checks None 
Docker-worker i-O3aibaach6fecaicO — 12. micro us-east-Tb @ running @ 22 checks... None 

@  —Docker-Manager i-O6f5103949862e4a 2.micro us-east-1b @ running © 2/2checks... None 


Instance: | i-061395103549862c4a (Docker-Manager) 


Description Status Checks 
Instance ID 


Instance siale 
Instance type 
Elastic IPs 
Availability zone 
Security groups 


Monitoring | Tags 


i-D6f95103949862c4a Public DMS (IP) 
running IPv4 Public IP 
L2. micro IFv& IPs 

Private ONS 
us-east-1b Private IPs 
Docker-Swarrm WideSG-L4215 VP ZELS , Secondary private IPs 


Docker-Managerv pc S -JOISRF 3XMSGY 


ew inbound rules 


Figure 8-5. EC2 instances for the Docker swarm 


Public ONS! ec2-54-B4-122-157.compute-1.amazoónaws.com 


gc2-54-B4-133-157 computa- 
1.amazonaws.com 
54 84. 133.157 


ip-172-31-25-121.&c2 internal 
172.31.25.121 


a af s 


o 0 0 


1to03013 


Public DNS (IPv4) 


ec 2-52-203-7 1-50. cam 
ec2-34-205-143-27 co.. 
gc 2-54-84-133- 157 co... 


F 


mmm:- 


SSH Login to the Swarm manager using the public IP address, which may be obtained from the EC2 
console, as shown in Figure 8-5. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.84.133.157 


Welcome to Docker! 
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List the nodes in the Swarm; three nodes should be listed. 


~ $ docker node Ils 

ID HOSTNAME STATUS AVAILABILITY MANAGER 
STATUS 

Owaa5g3b6j641xtwsygvjvwc1 ip-172-31-0-147.ec2.internal Ready Active 

e7vigin0luuo1kynjn133v9pa ip-172-31-29-67.ec2.internal Ready Active 

ptm7e0p346zwypos7wnpcm72d * ip-172-31-25-121.ec2.internal Ready Active Leader 


Creating and Scheduling a Service: The Spread Scheduling 


First, we discuss the default spread scheduling using a MySQL database service as an example. From the 
Swarm manager node, run the following command to create a five-replica service for MySQL. The output is 
the service ID (shown in italics). 


~ $ docker service create \ 
--env MYSQL ROOT _PASSWORD='mysql'\ 
--replicas 5 \ 
--name mysql \ 

mysql 

1onpemnoz4x1lh3sv5umab8uo 


> 
> 
> 
> 


Subsequently, list the services using docker service ls. Initially, the REPLICAS column could be 0/5, 
indicating that none of the replicas are scheduled and running yet. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
1onpemnoz4x1 mysql replicated 0/5 mysql:latest 


Run the command again after a while; all the replicas should be running as indicated by a 5/5 in the 
REPLICAS column. List the service replicas using the docker service ps mysql command. The tasks should 


be running or preparing to run. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

fwjbu3gt2zno mysql.1 mysql:latest ip-172-31-0-147.ec2.internal 
Running Preparing 8 seconds ago 

w0521ik1awjf mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Preparing 8 seconds ago 

z9wn2nrzfzt8 mysql.3 mysql:latest ip-172-31-0-147.ec2.internal 
Running Preparing 8 seconds ago 

tm8jbque3xbb mysql.4 mysql:latest ip-172-31-25-121.ec2.internal 
Running Preparing 8 seconds ago 

7drxfy3vbmp5 mysql.5 mysql:latest ip-172-31-29-67.ec2.internal 
Running Preparing 8 seconds ago 


Following the spread scheduling strategy, two of the replicas are listed as scheduled on one of the 
worker nodes, two on the other worker node, and one on the manager node. Because of the odd number of 
replicas, the placement cannot be completely evenly distributed, but a single node does not have more than 
two replicas. 
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To see how the spread scheduling strategy distributes the replicas evenly across a Swarm, scale the 
service to six replicas. The output of the docker service scale command is in italics. 


~ $ docker service scale mysql=6 
mysql scaled to 6 


Subsequently, list the replicas. Each node has two replicas scheduled on it, as the spread scheduling 
policy is designed to schedule. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


fwjbu3gt2zno 
Running 
w0521ik1awjf 
Running 
z9wn2nrzfzt8 
Running 
tm8jbque3xbb 
Running 
7drxfy3vbmp5 
Running 
ut jo8lwbtzf7 
Running 


NAME IMAGE 
CURRENT STATE 
mysql.1 mysql:latest 


Running 13 seconds ago 
mysql.2 mysql:latest 
Running 12 seconds ago 
mysql.3 mysql:latest 
Running 13 seconds ago 
mysql.4 mysql:latest 
Running 8 seconds ago 
mysql.5 mysql:latest 
Running 12 seconds ago 
mysql.6 mysql:latest 
Running 5 seconds ago 


NODE 

ERROR PORTS 
ip-172-31-0-147.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-0-147.ec2.internal 
ip-172-31-25-121.ec2.internal 


ip-172-31-29-67.ec2.internal 


ip-172-31-25-121.ec2.internal 


As a service replica or task is nothing but a slot to run a container, each node runs two containers for the 


mysql service. 


To further demonstrate spread scheduling, scale down the service to three tasks. The command output 


is in italics. 


~ $ docker service scale mysql-3 
mysql scaled to 3 


List the service tasks. Each node has one task running on it, which again is an evenly spread scheduling 


of tasks. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


w0521ik1awjf 
Running 
z9wn2nrzfzt8 
Running 
ut jo8lwbtzf7 
Running 


NAME IMAGE 
CURRENT STATE 
mysql.2 mysql:latest 


Running 40 seconds ago 
mysql.3 mysql:latest 
Running 41 seconds ago 
mysql.6 mysql:latest 
Running 33 seconds ago 


NODE 

ERROR PORTS 
ip-172-31-29-67.ec2.internal 
ip-172-31-0-147.ec2.internal 


ip-172-31-25-121.ec2.internal 
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Desired State Reconciliation 


When a service is created or is scaled up or down, the service initially has a discrepancy between the 
current state and the desired state. The different values for the desired state are ready, running, shutdown, 
and accepted. Docker services are designed for desired state reconciliation, which implies that the Swarm 
manager continuously monitors the cluster state to reconcile any differences between the desired state 
of a service and the current state. The current state of a task can be assigned, preparing, ready, running, 
shutdown, or pending. A task that has been assigned to a node but is not currently running is in the assigned 
state. A task that has desired state as running and is preparing to run is in the preparing current state. A task 
is in the pending state if no node in the Swarm can run the task. 

In the following task listing, some tasks have a desired state and current state of running. These tasks 
have reconciled their desired state. One task has a desired state set to running, but the current state is 
pending. Another task has a desired state set to shutdown and a current state set to assigned. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

opxf4ne7iyy6 mysql.1 mysql: latest ip-172-31-25-121.ec2.internal 
Running Running 9 minutes ago 

x30y3j1ea047 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 8 minutes ago 

w4ivsbvwqqzq mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 4 minutes ago 

j91p080ojof j7 mysql.4 mysql:latest 

Running Pending 28 seconds ago 

phizpsjsvp69 V mysql.4 — mysql:latest ip-172-31-7-137.ec2.internal 
Shutdown Assigned 33 seconds ago 

d3oxy6hxf jh3 V mysql.4 — mysql:latest ip-172-31-40-70.ec2.internal 
Shutdown Running 43 seconds ago 

ic331aasjpdm mysql.5 mysql:latest ip-172-31-44-104.ec2.internal 
Running Running 8 minutes ago 


In an earlier task listing, all tasks were in the current state preparing and the desired state running. 

Swarm mode is designed to reconcile the desired state as much as feasible, implying that if node 
resources are available, the desired number of replicas runs. To demonstrate, update the Docker for AWS 
CloudFormation stack by choosing Actions > Update Stack, as shown in Figure 8-6. 


Ü CloudFermation ~ 


Stacks 














5 o 
e Introducing StackSats 
AWS StackSel Is a container for a set af AWS CloudFommation stacks and allows you lo create stacks across multiple AVIS Accounts and ANS Regions. Open the Sracksets console và get 
stared 
cC ù 
Fitter: Active ~ Create Change Set For Current Stack Showing 1 stack 
Stack Name Status Description 
- Delete Stack m E E mmm 
x, Docker CREATE, COMPLETE Docker CE for AWS 17.06.0-ce (17.06.0-ce-awsz) 


ViewfEdi template in Designer 





Figure 8-6. Updating a stack 
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Decrease the number of worker nodes from two to one, as shown in Figure 8-7. 


(D CloudFormation ~ Stacks >+ Stack Detail » Update Stack 


Update Docker stack 


Select Template 
| Specity Details 
Options 


Review 


specify Details 


Specity parameter values, You can use or change the default parameter values, which are defined in the AWS CloudFormation template, Learn more, 


Stack name Docker 


Parameters 


Swarm Size 


Number of Swarm 
managers? 


Humber of Swarm worker | 1 
nodes? 


"| Murmnber of Swan manager nodes (1, 3, 5) 


Figure 8-7. Decreasing the number of worker nodes to one 


Subsequently, list the service replicas from the Swarm manager node. 


docker service ps mysql 


The service replicas running on the Swarm worker node that was made to leave the Swarm are listed as 
shutdown. New replicas are started on the remaining two nodes in the Swarm to reconcile the desired state. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


pi4bbk7ijimt 
Running 
w0521ik1awjf 
Running 
uatsaay7axlc 
Running 
z9wn2nrzfzt8 
Shutdown 
w1tlwOfom42q 
Running 
qc75buhzzct3 
Shutdown 
sO09ts9s8np3d 
Running 
ut jo8lwbtzf7 
Running 


NAME IMAGE 
CURRENT STATE 

mysql.1 mysql:latest 
Running 5 minutes ago 

mysql.2 mysql:latest 
Running 7 minutes ago 

mysql.3 mysql:latest 


Running about a minute ago 
V mysgl.3 mysql:latest 
Running 2 minutes ago 
mysql.4 mysql:latest 
Running about a minute ago 
V mysql.4 — mysql:latest 
Running 2 minutes ago 


mysql.5 mysql:latest 
Running 5 minutes ago 
mysql.6 mysql:latest 


Running 7 minutes ago 


NODE 

ERROR PORTS 
ip-172-31-29-67.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-25-121.ec2.internal 
Owaa5g3b6j641xtwsygvjvwc1 
ip-172-31-29-67.ec2.internal 
Owaa5g3b6j641xtwsygv jvwc1 


ip-172-31-25-121.ec2.internal 


ip-172-31-25-121.ec2.internal 
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Listing only the replicas with a desired state of running, the six replicas are listed as scheduled evenly 
between the two nodes—three replicas on the manager node and three replicas on the worker node. 


~ $ docker service ps -f 


ID 


DESIRED STATE 


pi4bbk7ijimt 
Running 
w0521ikiawjf 
Running 
uatsaay7axlc 
Running 
w1tlwOfomA42q 
Running 
s09ts9s8np3d 
Running 
utjo8lwbtzf7 
Running 


NAME 


CURRENT 


mysql.1 
Running 
mysql.2 
Running 
mysql.3 
Running 
mysql.4 
Running 
mysql.5 
Running 
mysql.6 
Running 


desired-state-running mysql 


IMAGE 


STATE 


6 


8 


2 


2 


6 


8 


mysql: latest 
minutes ago 
mysql:latest 
minutes ago 
mysql:latest 
minutes ago 
mysql:latest 
minutes ago 
mysql:latest 
minutes ago 
mysql: latest 
minutes ago 


ERROR 


NODE 

PORTS 
ip-172-31-29-67.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-25-121.ec2.internal 
ip-172-31-29-67.ec2.internal 


ip-172-31-25-121.ec2.internal 


ip-172-31-25-121.ec2.internal 


The spread scheduling strategy does not reschedule already running replicas to achieve even spread 
across a Swarm if new nodes are added to the Swarm. To demonstrate this, we increase the number of 


worker nodes back to two, as shown in Figure 8-8. 


Update Docker stack 


Select Template 
| Specify Details 
Options 


Review 


Figure 8-8. Re-adding a worker node to Swarm 


Specify Details 


Specify parameter values. You can use or change the default parameter values, which are defined in the AWS CloudFonmation template. Learn more. 


Stack name 


Docker 


Parameters 


Swarm Size 


Number of Swarm 1 


managers? 


Humber of Swarm worker 


nodes? 


Adding a node to a swarm does not shut down replicas on other nodes and start replicas on the new 
node. Listing the running replicas does not indicate a replacement of the service replicas. Service replicas 
continue to run on the nodes they were running on before the new node was added—three on the manager 
node and three on the worker node. 
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ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

p14bbk7ij1mt mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 15 minutes ago 

w0521ik1awjf mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 17 minutes ago 

uatsaay7axlc mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 12 minutes ago 

z9wn2nrzfzt8 V mysql.3 | mysql:latest Owaa5g3b6j641xtwsygv jvwc1 
Shutdown Running 13 minutes ago 

witlwo0fom42q mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 12 minutes ago 

qc75buhzzct3 V mysql.4 — mysql:latest Owaa5g3b6j641xtwsygv jvwc1 
Shutdown Running 13 minutes ago 

s09ts9s8np3d mysql.5 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 15 minutes ago 

utjo8lwbtzf7 mysql.6 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 17 minutes ago 


scheduling Tasks Limited by Node Resource Capacity 


The scheduling policy is limited by the available node resources, implying that service replicas cannot 

be made to run if not enough node resources in terms of CPU and memory are available. Resource usage 
cannot exceed node capacity. The replicas are still allocated to the service to define the desired state but may 
not be running due to insufficient resources. To demonstrate this, we remove the service mysql and create 
the service again with the specified resource requests and limits. Command outputs are shown in italics. 


~ $ docker service rm mysql 
mysql 


~ $ docker service create ^ 

--env MYSQL ROOT PASSWORD-'mysql'^ 

--replicas 1 \ 

--name mysql \ 

--reserve-cpu 1 --limit-cpu 2 --reserve-memory 256mb --limit-memory 512mb mysql 
Oge2thyOdlviroli6k8thist1 


> 
> 
> 
> 


Listing the services indicates that one replica of the service is created. 


~ $ docker service ls 
ID NAME 
Oge2thyodlvi mysql 


IMAGE 
mysql:latest 


MODE REPLICAS PORTS 


replicated 1/1 


The single replica is scheduled on the manager node, which is chosen randomly if all nodes in a Swarm 
have the same node ranking. 
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~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
opxf4ne7iyy6 mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 8 seconds ago 


Next, to potentially make the service replicas consume more resources than available, scale the service 
to five replicas. 


~ $ docker service scale mysql=5 
mysql scaled to 5 


Listing the services indicates that 3/5 Replicas are running. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
Ooge2thyodlvi mysql replicated 3/5 mysql:latest 


Listing the service replicas indicates that some of the replicas are pending instead of running. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

opxf4ne7iyy6 mysql.1 mysql: latest ip-172-31-25-121.ec2.internal 
Running Running 4 minutes ago 

x30y3j1ea047 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 3 minutes ago 

w4ivsbvwqqzq mysql.3 mysql:latest 

Running Pending 3 minutes ago 

d3oxy6hxftjh3 mysql.4 mysql:latest 

Running Pending 3 minutes ago 

1c331aasjpdm mysql.5 mysql:latest ip-172-31-44-104.ec2.internal 
Running Running 3 minutes ago 


The pending state implies that the replicas are allocated to the service but not scheduled on any node 
yet. Only three replicas could run based on the requested resources and available node resources, one on 
each node. 

Because the replicas are not scheduled due to lack of resources, we add one or more new worker nodes 
to potentially schedule the replicas to reconcile the desired state. Increase the number of worker nodes to 
five, as shown in Figure 8-9. 
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Select Template Specify Details 
| Specify Details 
Options Specify parameter values. You can use or change the default parameter values, which are defined in the AWS CloudFormation template. Learn more 
MV PUY Jj 
Review 
Stack name Docker 
Parameters 
Swarm Size 


Number of Swarm 
managers? 


ls Number of Swarm worker 
, nodes? 


Figure 8-9. Increasing the number of worker nodes to five 


The Swarm should list six nodes after a new node is added. As resources became available for the 
pending tasks, the tasks get scheduled and start running. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
opxf4ne7iyy6 mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 5 minutes ago 

x30y3j1ea047 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 

Running Running 4 minutes ago 

w4ivsbvwqqzq mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 

Running Running 21 seconds ago 

d3oxy6hxftjh3 mysql.4 mysql:latest ip-172-31-40-70.ec2.internal 

Running Preparing 30 seconds ago 

ic331aasjpdm mysql.5 mysql:latest ip-172-31-44-104.ec2.internal 
Running Running 4 minutes ago 


If the number of worker nodes is decreased, some of the tasks are descheduled, as indicated by the 
shutdown desired state. 
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~ $ docker service ps mysql 


ID 

DESIRED STATE 
opxf4ne7iyy6 
Running 
x30y3j1ea047 
Running 
w4ivsbvwqqzq 
Running 
j91p080jof j7 
Running 
phizpsjsvp69 
Shutdown 
d3oxy6hxf jh3 
Shutdown 
1c331aasjpdm 
Running 


NAME IMAGE 
CURRENT STATE 

mysql.1 mysql:latest 
Running 9 minutes ago 

mysql.2 mysql:latest 
Running 8 minutes ago 

mysql.3 mysql:latest 
Running 4 minutes ago 

mysql.4 mysql:latest 


Pending 28 seconds ago 

V mysql.4 — mysql:latest 
Assigned 33 seconds ago 
V mysql.4 — mysql:latest 
Running 43 seconds ago 
mysql.5 mysql:latest 
Running 8 minutes ago 


NODE 

ERROR PORTS 
ip-172-31-25-121.ec2.internal 
ip-172-31-29-67.ec2.internal 


ip-172-31-2-177.ec2.internal 


ip-172-31-7-137.ec2.internal 
ip-172-31-40-70.ec2.internal 


ip-172-31-44-104.ec2.internal 


Updating the service to lower CPU and memory resource usage reserved only updates the 
UpdateConfig for the service. This does not lower the resource usage ofthe already running tasks or 
make pending or shutdown tasks run. As an example, lower the resource reserves and limits for the mysql 
service when some of the tasks are pending or shutdown due to lack of resources. 


~ $ docker service update --reserve-cpu .1 --limit-cpu .5 --reserve-memory  64mb 
--limit-memory 128mb mysql 


mysql 


The UpdateConfig gets modified, but only applies to new replicas created after that point. 


~ $ docker service inspect mysql 


l 


) 
"Resources": ( 
"Limits": { 
"NanoCPUs": 500000000, 
"MemoryBytes": 134217728 
3 
"Reservations": { 
"NanoCPUs": 100000000, 
"MemoryBytes": 67108864 
) 


Only three of the replicas in the mysql service are actually running. 
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~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

opxf4ne7iyy6 mysql.1 mysql: latest ip-172-31-25-121.ec2.internal 
Running Running 10 minutes ago 

x30y3j1ea047 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 10 minutes ago 

w4ivsbvwqqzq mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 5 minutes ago 

rm9uj4gevt5b mysql.5 mysql:latest 

Running Pending 33 seconds ago 


To force the service tasks to use the new resource settings, scale down the service to one task and then 
scale back up to five tasks. 


~ $ docker service scale mysql=1 
mysql scaled to 1 
~ $ docker service scale mysql-5 
mysql scaled to 5 


All five tasks are now running. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

anai3mptbnkp mysql.1 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 17 seconds ago 

opxf4ne7iyy6 V mysql.1 = mysql:latest ip-172-31-25-121.ec2.internal 
Shutdown Shutdown 18 seconds ago 

Imkn8150t334 mysql.2 mysql: latest ip-172-31-25-121.ec2.internal 
Running Running 10 seconds ago 

7uz7q86wnzn4 mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 11 seconds ago 

ubh4m39aw8m9 mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 11 seconds ago 

56pnrzajogvs mysql.5 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 10 seconds ago 


Adding Service Scheduling Constraints 


Docker Swarm supports placement or scheduling constraints for scheduling new tasks. Service placement 
constraints are additional criteria for placement of service tasks and could be based on node attributes, 
metadata, and engine metadata. The Swarm scheduler uses the following sequence to schedule a service task. 


1. Doesthe node satisfy all the placement constraints? 
2. Does a node meet the scheduling policy requirements of an even spread? 


3. Does the node have sufficient resources to schedule a task? 
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A placement constraint may be added using the --constraint option with the docker service create 
command. For an already running service, constraints may be added and removed with the - -constraint-add 
and --constraint-rm options, respectively, with the docker service update command. The node 
attributes discussed in Table 8-1 may be used to specify constraints. 


Table 8-1. Node Attributes for Constraints 
Node Attribute Description Example 


node.id Specifies the node ID. Node IDs are listed using the node.id==a3r56hj7y 
docker node ls command. 


node.hostname Specifies the node’s hostname. The node's hostname _node. hostname! =ip-10-0-0- 


is listed with the docker node ls command. ec2.internal 
node.role Specifies the node role, which is one of worker or node.role--manager 
manager. 
node.labels Specifies the node labels added by a user. A label node.labels.db--mysql 


is a key-value pair. When adding a node label, the 
node.labels. prefix is to be omitted and gets added 
automatically. Adding and using node labels is 
discussed in a subsequent section. 


engine.labels Docker Engine labels such as drivers, operating engine.labels.os--coreos 
system, version. 


Next, we discuss some examples of using scheduling constraints. 


Scheduling on a Specific Node 


In this section we schedule service replicas on specific nodes in a Swarm. List the node IDs with the docker 
node ls command. The Swarm has the following three nodes available for scheduling. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER 
STATUS 

8ih6uvu8ugdemnovzkg6v7mzg ip-172-31-2-177.ec2.internal Ready Active 

e7viginOluuoikynjnl33v9pa ip-172-31-29-67.ec2.internal Ready Active 

ptm7e0p346zwypos7wnpcm72d * ip-172-31-25-121.ec2.internal Ready Active Leader 


We can schedule a service by node role. Create a mysql service with the placement constraint that the 
service tasks be scheduled on worker nodes only. First, remove the mysql service if it’s already running 


~ $ docker service rm mysql 
mysql 
~ $ docker service create \ 


>  --env MYSQL ROOT PASSWORD='mysql'\ 
>  --replicas 3 \ 

>  --constraint node.role--worker V 

>  --name mysql \ 

» mysql 


nzgte4zacix8itx6t98y5gi42 
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The service is created and three tasks are scheduled only on the two worker nodes, as listed in the 
running service tasks. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

f5ti5mnrftoh mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 19 seconds ago 

oxvq4ljuq6yz mysql.2 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 19 seconds ago 

k5jo8621vsxf mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 19 seconds ago 


Next, we use the node ID to schedule a service's tasks. Copy the node ID for the manager node, which 
is also the leader in the Swarm being the only manager node. Substitute the node ID in the following 
command to create a service for the MySQL database and schedule replicas only on the manager node. 


docker service create \ 
--env MYSQL ROOT PASSWORD=' mysql’ \ 
--replicas 3 \ 
--constraint node.id ==<nodeid> 
--name mysql \ 

mysql 


A service is created with three tasks. Command output is shown in italics. 


~ ¢ docker service create \ 


>  --env MYSQL ROOT PASSWORD='mysql' \ 

>  --replicas 3 \ 

>  --constraint node. id==ptm7e0p346zwypos7wnpcm72d\ 
>  --name mysql ^ 

» mysql 


uiqi6zqnch9hn7x6k516axg7h 
All the three replicas of the service are scheduled on the manager node only. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

lbttu95qdjvy mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 21 seconds ago 

89x0z940nO0fb mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 21 seconds ago 

3s6508aimdaj mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 22 seconds ago 


147 


CHAPTER 8 ~ SCHEDULING 


Adding Multiple Scheduling Constraints 


Multiple node constraints may also be specified and every constraint expression must be met using AND 
for the scheduler to schedule a replica on a node. As an example, we create a service with two roles, one 
that constrains the node role to worker and the other constrains the node hostname not to be a specific 
hostname ip-172-31-2-177.ec2.internal. 


~ $ docker service create \ 

>  --env MYSQL ROOT PASSWORD='mysql'\ 

>  --replicas 3 \ 

>  --constraint node.role--worker \ 

>  --constraint node.hostname!=ip-172-31-2-177.ec2.internal\ 
>  --name mysql ^ 

» mysql 

87g0c8kauhz8yb4wv2ryc2vqr 


A service gets created. Listing the services lists 3/3 replicas as running. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
87g0c8kauhz8 mysql replicated 3/3 mysql:latest 
Listing the service tasks indicates that all tasks are scheduled on a single worker node. The two 
constraints are met: the node is a worker node and not the worker node with hostname ip-172-31-2-177.ec2. 


internal. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

jlfk79mb6m6a mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 13 seconds ago 

if5y39ky884q mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 13 seconds ago 

zctm6mzblAdu mysql.3 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 13 seconds ago 


If the mysql service is updated to remove the constraints, the spread scheduling strategy reschedules the 
tasks based on node ranking. As an example, update the service to remove the two placement constraints 
added. A constraint is removed with the -constraint-rm option ofthe docker service update command. 


~ $ docker service update \ 


>  --constraint-rm node.role--worker \ 

>  --constraint-rm node.hostname! =ip-172-31-2-177.ec2.internal\ 
> mysql 

mysql 


When a service is updated to remove constraints, all the service tasks are shut down and new service 
tasks are started. The new service tasks are started, one each on the three nodes in the Swarm. 
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~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
d22bkgteivot mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Ready Ready less than a second ago 


jlfk79mb6m6a V mysql.1 — mysql:latest ip-172-31-29-67.ec2.internal 
Shutdown Running 1 second ago 


mp757499j3io mysql.2 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 1 second ago 

if5y39ky884q V mysql.2 = mysql:latest ip-172-31-29-67.ec2. internal 
Shutdown Shutdown 2 seconds ago 

jtdxuctebofl mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 4 seconds ago 

zctm6mzblAdu V mysql.3 | mysql:latest ip-172-31-29-67.ec2.internal 
Shutdown Shutdown 5 seconds ago 


List only the running tasks. One task is listed running on each node. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

d22bkgteivot mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 46 seconds ago 

mp757499j3io mysql.2 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 49 seconds ago 

jtdxuctebofl mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 53 seconds ago 


Similarly, multiple node constraints could be used to run replicas only on a manager node. Next, we 
update the mysql service to run on a specific manager node. First, promote one of the worker nodes to 
manager. 


~ $ docker node promote ip-172-31-2-177.ec2.internal 
Node ip-172-31-2-177.ec2.internal promoted to a manager in the swarm. 


Subsequently, two manager nodes are listed as indicated by the Manager Status for two of the nodes. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER 
STATUS 
8ih6uvu8ugdemnovzkg6v7mzg ip-172-31-2-177.ec2.internal Ready Active Reachable 
e7viginOluuoikynjnl33v9pa ip-172-31-29-67.ec2.internal Ready Active 
ptm7eOp346zwypos7wnpcm72d * ip-172-31-25-121.ec2.internal Ready Active Leader 


Update the mysql service to add multiple node constraints to run replicas only on a specific manager 
node. Constraints are added using the --constraint-add option ofthe docker service update command. 


~ $ docker service update \ 


>  --constraint-add node.role==manager \ 

>  --constraint-add node.hostname==ip-172-31-2-177.ec2.internal\ 
> mysql 

mysql 
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Again, all service tasks are shut down and new tasks are started, all on the specified manager node that 
was promoted from the worker node. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

eghm10r6yg5g mysql.1 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 28 seconds ago 

bhfngac5ssm7 mysql.2 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 22 seconds ago 

ts3fgvq9000s mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 25 seconds ago 


Adding Node Labels for Scheduling 


Next, we discuss how node labels can be used to specify service placement constraints. Labels may be 
added to a node with the following command syntax, in which variables are <LABELKEY>, <LABELVALUE>, and 
«NODE». The «NODE» is the node ID or hostname. 


docker node update --label-add  «LABELKEY»-«LABELVALUE» «NODE» 


As an example, add the label db-mysq1 to the node with a hostname set to ip-172-31-25-121.ec2. 
internal, which is the leader node. 


~ $ docker node update --label-add db=mysql ip-172-31-25-121.ec2.internal 
ip-172-31-25-121.ec2.internal 


A node label is added. On inspecting the node, the label is listed in the Labels field. 


~ $ docker node inspect ip-172-31-25-121.ec2.internal 
[ 
"Spec": { 
"Labels": { 
"db": "mysql" 
Js 


"Role": "manager", 
"Availability": "active" 


Js 


Next, create a service that uses the node label to add a placement constraint. The --constraint option 
for the label must include the prefix node. labels. 


~ $ docker service rm mysql 
mysql 
~ $ docker service create \ 


>  --env MYSQL ROOT PASSWORD='mysql' \ 

>  --replicas 3 \ 

>  --constraint node.labels.db--mysql V 
>  --name mysql \ 

» mysql 


2hhccmj9senseazbet11dekoa 
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The service is created. Listing the tasks lists all the tasks on the Leader manager node, which is what the 
node label constraint specified. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

g5jz9im3fufv mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 18 seconds ago 

bupr27bs57h1 mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 18 seconds ago 

5bb2yf8aehqn mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 18 seconds ago 


The label added may be removed with the - - label-rmoption of the docker node update command in 
which the only the label key is specified. 


docker node update --label-rm db  ip-172-31-25-121.ec2.internal 


Adding, Updating, and Removing Service Scheduling Constraints 


In an earlier section, we discussed adding placement constraints when creating a service with docker 
service create. Placement constraints may be added/removed with the docker service update 
command using the - -constraint-add and --constraint-rm options. To discuss an example of updating 
placement constraints, we create a mysql service with three replicas and no placement constraints to 

start with. 


~ $ docker service rm mysql 

mysql 

~ $ docker service create \ 

>  --env MYSQL ROOT PASSWORD='mysql'\ 
>  --replicas 3 \ 

>  --name mysql ^ 

» mysql 

az3cq6sxwrrk4mxkksdu211i25 


A mysql service gets created with three replicas scheduled on the three nodes in the Swarm, using the 
spread policy. 

Next, update the service with the docker service update command to add a constraint for the service 
replicas to run only on the manager nodes. 


~ $ docker service update \ 

> --constraint-add node.role--manager ^ 
> mysql 

mysql 


In a Swarm with two manager nodes, all the service tasks are shut down and new tasks are started only 
on the manager nodes. 
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~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
pjwseruvy4rj mysql.1 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 4 seconds ago 

s66g9stz9af5 V mysql.1 — mysql:latest ip-172-31-2-177.ec2.internal 
Shutdown Shutdown 4 seconds ago 

yqco9zd0vq79 mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 9 seconds ago 

8muu6gbghhnd V mysql.2  mysql:latest ip-172-31-25-121.ec2.internal 
Shutdown Shutdown 10 seconds ago 

8x7xlavcxdau mysql.3 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 7 seconds ago 

qx95vwi2h547 X mysql.3 — mysgl:latest ip-172-31-29-67.ec2.internal 
Shutdown Shutdown 7 seconds ago 


Scheduling constraints may be added and removed in the same docker service update command. 
As an example, remove the constraint for the node to be a manager and add a constraint for the node to be a 
worker. 


~ $ docker service update \ 


>  --constraint-rm node.role==manager ^ 
>  --constraint-add node.role--worker ^ 
» mysql 

mysql 


Again. all the service tasks are shut down and new tasks are started only on the worker nodes. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

6ppgmvw91v75 mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 9 seconds ago 

qmoloki65v9s mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 17 seconds ago 


yplotcift920 mysql.3 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 


If the only scheduling constraint that specifies the node role as worker is removed, the spread 
scheduling strategy starts new tasks spread evenly across the Swarm. To demonstrate, remove the constraint 


for the node role to be a worker. 


~ $ docker service update --constraint-rm node.role--worker mysql 
mysql 


Subsequently, new tasks are spread across the nodes in the Swarm. 
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~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

jpx4jjw6lods mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 5 seconds ago 

ngajiikihugb mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 12 seconds ago 

40eaujzlux88 mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 8 seconds ago 


spread Scheduling and Global Services 


A global service runs one task on every node in a Swarm. A global service cannot be scaled to create more/ 
fewer tasks. As a result, the spread scheduling policy concept does not apply to global services. However, 
node constraints may be applied to global services. As an example, we create a global service for the mysql 
database. Apply a placement constraint that the service should be available only on worker nodes. 


~ $ docker service create \ 


> --mode global \ 

>  --env MYSQL ROOT PASSWORD-'mysql'* 
>  --constraint node.role--worker \ 

>  --name mysql ^ 

> mysql 


jtzcwatpoo1q9r26niuubd8me 


The global service is created. Listing the service tasks for the tasks with desired state as running lists 
only the tasks on the worker nodes. 


~ $ docker service ps -f desired-state-running mysql 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

o5nskzpv27j9 mysgl.e7viginOluuoikynjnl33v9pa mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 17 seconds ago 


If created without the constraint to schedule on worker nodes only, a global service schedules one task 
on each node, as demonstrated by the following example. 


~ $ docker service rm mysql 

mysql 

~ $ docker service create ^ 

> --mode global \ 

>  --env MYSQL ROOT PASSWORD='mysql' \ 
>  --name mysql ^ 

» mysql 

mv9yzyyntdhzz41zssbutcsvw 
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~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

mc87btddhmpl  mysql.e7viginOluuoikynjnl33v9pa mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 19 seconds ago 


oOwfdq9sd8yt  mysql.ptm7eOp346zwypos7wnpcm72d mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 19 seconds ago 
wt2q5k2dhqjt mysql.81h6uvu8uqoemnovzkg6v7mzg mysgl:latest ip-172-31-2-177.ec2.internal 
Running Running 19 seconds ago 


Summary 


This chapter discussed the scheduling policy of spread used in the Docker Swarm mode, whereby service 
replicas are spread evenly across nodes in a Swarm based on node ranking; a higher node ranking gets a 
service replica placement priority. We also discussed the effect of limited node resource capacity and how 
to alleviate it by adding new nodes to the Swarm. We discussed placement constraints for scheduling new 
replicas. The spread scheduling policy is not relevant for global services as they create one service task on 
each node by default. However, scheduling constraints may be used with global services. In the next chapter 
we discuss rolling updates to Docker services. 
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CHAPTER 9 


Rolling Updates 





The Docker Swarm mode provisions services consisting of replicas that run across the nodes in the Swarm. 
A service definition is created when a service is first created/defined. A service definition is created with the 
docker service create command. That command provides several options, including those for adding 
placement constraints, container labels, service labels, DNS options, environment variables, resource 
reserves and limits, logging driver, mounts, number of replicas, restart condition and delay, update delay, 
failure action, max failure ratio, and parallelism, most of which were discussed in Chapter 4. 


The Problem 


Once a service definition has been created, it may be required to update some of the service options such as 
increase/decrease the number of replicas, add/remove placement constraints, update resource reserves and 
limits, add/remove mounts, add/remove environment variables, add/remove container and service labels, 
add/remove DNS options, and modify restart and update parameters. If a service is required to be shut down 
as a whole to update service definition options, an interruption of service is the result. 


The Solution 


Docker Swarm mode includes the provision for rolling updates. In a rolling update, the service is not shut 
down, but individual replicas/tasks in the service are shut down one at a time and new service replicas/ 
tasks based on the new service definition are started one at a time, as illustrated in Figure 9-1. As a result the 
service continues to be available during the rolling update. The service tasks that are served to a client could 
be from both old and new service definitions during a rolling update. As an example, if the rolling update 
performs an update to a more recent image tag, some of the tasks served to external clients during the rolling 
update could be from a mix of old image tag and new image tag. 
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Rolling Update Service 


mysql 


Service 
mysql 


Seivice 
eplicas 


Figure 9-1. Rolling update 


Rolling update creates a new service definition and a new desired state for a service. Rolling update 
involves shutting down all service replicas and starting all new service replicas and does not apply to service 
replicas that have not yet been scheduled, due to lack of resources for example. Even updating just the 
number of replicas in a rolling update shuts down or fails all the old replicas and starts all new replicas. 

The following sequence is used by the scheduler during a rolling update. 


1. Thefirst task is stopped. 

2. An update for the stopped task is scheduled. 

3. A Docker container for the updated task is started. 

4. Ifthe update to a task returns RUNNING, wait for the duration specified in 


- -update-delay and start the update to the next task. 
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5. Ifduring the update, a task returns FAILED, perform the --update-failure- 
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action, which is to pause the update by default. 


6. Restart a paused update with docker service update <SERVICE-ID>. 


7. Ifan update failure is repeated, find the cause of the failure and reconfigure the 
service by supplying other options to the docker service update. 


Setting the Environment 


Create a Docker Swarm consisting of a manager node and two worker nodes using Docker for AWS, as 
discussed in Chapter 3. Obtain the public IP address of the manager instance from the EC2 console and then 


SSH login to the instance. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.84.133.157 


Welcome to Docker! 
List the Swarm nodes. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
81h6uvu8uqOemnovzkg6v7mzg ^ ip-172-31-2-177.ec2.internal Ready Active 
e7viginOluuoikynjnl33v9pa | ip-172-31-29-67.ec2.internal Ready Active 
ptm7e0p346zwypos7wnpcm72d * ip-172-31-25-121.ec2.internal Ready Active 


Creating a Service with a Rolling Update Policy 


A rolling update policy or update config consists of the service definition options discussed in Table 9-1. 


Table 9-1. Rolling Update Options 
Option 
--update-delay 


--update-failure-action 


--update-max-failure-ratio 


--update-monitor 


--update-parallelism 


Description 


Delay between updates (ns|us|ms|s|m|h). 


Action on update failure. Value may be pause 
or continue. 


Duration after each task update to monitor for 
failure (ns|us|ms|s|m|h). 


Maximum number of tasks updated 


simultaneously. A value of 0 updates all at once. 


Leader 


Default Value 


0 seconds 


pause 


0 seconds 
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To configure the rolling update policy at service deployment time, the options to be configured must 
be supplied when the service is created. As an example, create a service for MySQL database and specify the 
update policy options - -update-delay and --update-parallelism. 


~ $ docker service create \ 

>  --env MYSQL ROOT PASSWORD='mysql' \ 
>  --replicas 1 \ 

>  --name mysql ^ 

>  --update-delay 10s \ 

>  --update-parallelism 1 \ 

» mysql:5.6 

wr0z48viuguk1c40pa42ywrpn 


The service is created. Listing the services may not list all replicas as running initially, as indicated by 
0/1 in the REPLICAS column. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
wr0z48viuguk mysql replicated 0/1 mysql:5.6 


Running the same command after a while should list all replicas as running, as indicated by 1/1 in 
REPLICAS column. 


~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
wrOz48viuguk mysql replicated 1/1 mysql:5.6 


The single service replica is scheduled on the manager node itself and the Docker container for the 
replica is started. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

38dm9gm6cmvk mysql.1 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 13 seconds ago 


Creating a service using rolling update options does not by itself demonstrate a rolling update. It only 
defines the UpdateConfig settings of the service. In the next section we perform a rolling update. 


Rolling Update to Increase the Number of Replicas 


A rolling update could be used to update the number of replicas with the - - replicas option to the docker 
service update command. A rolling update updates the UpdateConfig policy applied when the service 

is first deployed. Next, we update the number of replicas for the mysq1:5.6 image based service from the 
one replica created in the preceding section. Run the following command to update the service definition 
to five replicas from one replica. The - -update-delay and --update-parallelism options modify the 
UpdateConfig of the service definition. The docker service update command outputs the service name if 
the update is successful. 
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~ $ docker service update \ 


>  --replicas 5 \ 

>  --update-delay 20s \ 

>  --update-parallelism 1 \ 
> mysql 

mysql 


Subsequently, the services listing may list some of the replicas as not started yet in the output to the 
docker service ls command. But, running the command again after a while should list all replicas as running. 


" $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
wr0z48viuguk mysql replicated 5/5 mysql:5.6 


During the rolling update, all the running tasks are shut down and new tasks are started. The desired 
state of the mysql.1 task gets updated to shutdown and the current state is set to failed. A new task mysql.1 
is started. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
ydqj6vf9rsgw mysql.1 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 26 seconds ago 

38dm9gm6cmvk V mysql.1 mysql:5.6 ip-172-31-25-121.ec2.internal 
Shutdown Failed 31 seconds ago "task: non-zero exit (137)" 

7bns96iu8ygz mysql.2 mysql:5.6 ip-172-31-29-67.ec2.internal 
Running Running 32 seconds ago 

62wfdbcv3cr4 mysql.3 mysql:5.6 ip-172-31-2-177.ec2.internal 
Running Running 33 seconds ago 

q166z5x0a21f mysql.4 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 14 seconds ago 

3n3b1j7ey732 V mysql.4 mysql:5.6 ip-172-31-25-121.ec2.internal 
Shutdown Failed 19 seconds ago "task: non-zero exit (137)" 

b11365y60vuu mysql.5 mysql:5.6 ip-172-31-2-177.ec2.internal 
Running Running 33 seconds ago 


When scaling from one to five replicas, first a few new tasks are started and then the task running 
initially is shut down so that the service continues to be available during the rolling update. If the only task in 
the service were to be shut down first before starting any new tasks, the service wouldn't have any running 
tasks for a short while. 

The desired state of running five replicas is not immediately reconciled during a rolling update. Fewer 
than five tasks could be running while the rolling update is in progress. Listing the running service tasks lists 
only three tasks as running. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

ydqj6vf9rsgw mysql.1 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 35 seconds ago 

7bns96iu8ygz mysql.2 mysql:5.6 ip-172-31-29-67.ec2.internal 
Running Running 40 seconds ago 

q166z5x0a21f mysql.4 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 22 seconds ago 
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When the rolling update has completed, five tasks are running. 


~ $ docker service ps -f desired-state-running mysql 


ID 

DESIRED STAT 
u8falo7q95cq 
Running 
luabknwzwqoj 
Running 
ce4l2qvtcanv 
Running 
iw8vwsxq3tjz 
Running 
qfi5fionjt2v 
Running 


E 


NAME IMAGE 
CURRENT STATE 


mysql.1 mysql:5.6 


Running 20 seconds ago 


mysql.2 mysql:5.6 


Running 13 seconds ago 


mysql.3 mysql:5.6 


Running 25 seconds ago 


mysql.4 mysql:5.6 


Running 6 seconds ago 


mysql.5 mysql:5.6 


Running 25 seconds ago 


NODE 

PORTS 
ip-172-31-25-121.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-2-177.ec2.internal 


ip-172-31-25-121.ec2.internal 


ip-172-31-29-67.ec2.internal 


Inspecting the service should list the updated number of replicas. The UpdateConfig is also listed with 
the docker service inspect command. 


~ $ docker service inspect mysql 
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"id 

"Name": "mysql", 

) 

"Mode": ( 
"Replicated": { 

"Replicas": 5 

} 

) 


"UpdateConfig": { 


Jy 


"Parallelism": 1, 
"Delay": 20000000000, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": O, 
"Order": "stop-first" 


"RollbackConfig": { 


Js 


"Parallelism": 1, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": O, 
"Order": "stop-first" 
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Rolling Update to a Different Image Tag 


A use case for a rolling update is to update to a newer image tag. As an example, perform a rolling update to 
update to Docker image mysql: latest from mysq1:5.6 for the mysql service. Update parallelismis set to 2 
to update two replicas at a time. 


~ $ docker service update --image mysql:latest --update-parallelism 2 mysql 


mysql 


The service rolling update gets started. Listing the service replicas lists nysq1:5.6 image-based replicas 
as shutting down, as indicated by the shutdown desired state and mysql: latest image-based replicas as 
starting, as indicated by the running desired state. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


vqc6rhzw5uxz 
Ready 
80kswuu4d5gc 
Shutdown 
u8falo7q95cq 
Shutdown 
ydqj6vf9rsgw 
Shutdown 
38dm9gm6cmvk 
Shutdown 
tvxjmahyo8uh 
Running 
luabknwzwqoj 
Shutdown 
7bns96iu8ygz 
Shutdown 
u2ea4xq4yx6t 
Running 
ce4l2qvtcanv 
Shutdown 
62wfdbcv3cr4 
Shutdown 
iw8vwsxq3tjz 
Running 
gl66z5x0a21f 
Shutdown 
3n3b1j 7ey732 
Shutdown 
f5vcf9mgluge 
Running 
qfi5fionjt2v 
Shutdown 
b11365y60vuu 
Shutdown 


NAME IMAGE 
CURRENT STATE 
mysql.1 mysql: 
Ready 7 seconds ago 

V mysql.1 mysql: 
Running 7 seconds ago 

V mysql.1 mysql: 
Failed 12 seconds ago 

V mysql.1 mysql: 


Failed 56 seconds ago 
V mysql.1 
Failed about a minute ago 
mysql.2 
Running 2 seconds ago 


V mysql.2 mysql: 
Failed 8 seconds ago 
V mysql.2 mysql: 


Failed 50 seconds ago 


mysql: 


mysql: 


ERROR 
latest 


5.6 


5.6 
"task: 
5.6 
"task: 
5.6 
"task: 
5.6 


5.6 
"task: 
5.6 


NODE 
PORTS 
ip-172-31-2-177.ec2.internal 


ip-172-31-2-177.ec2.internal 


ip-172-31-25-121.ec2.internal 
non-zero exit (1)" 
ip-172-31-25-121.ec2.internal 
non-zero exit (1)" 
ip-172-31-25-121.ec2.internal 
non-zero exit (137)" 
ip-172-31-29-67.ec2.internal 


ip-172-31-29-67.ec2.internal 
non-zero exit (137)" 
ip-172-31-29-67.ec2.internal 


"task: non-zero exit (137)" 


mysql.3 mysql:latest 
Running 4 seconds ago 

V mysql.3 mysql:5.6 
Shutdown 4 seconds ago 

V mysql.3 mysql:5.6 
Failed about a minute ago ‘task 
mysql.4 mysql:5.6 
Running 37 seconds ago 

V mysql.4 mysql:5.6 
Failed 43 seconds ago "task 
V mysql.4 mysql:5.6 
Failed about a minute ago ‘task 
mysql.5 mysql:5.6 
Running 14 seconds ago 

V mysql.5 mysql:5.6 
Failed 19 seconds ago "task 
V mysql.5 mysql:5.6 
Failed about a minute ago ‘task 


ip-172-31-2-177.ec2.internal 
ip-172-31-2-177.ec2.internal 


ip-172-31-2-177.ec2.internal 


: non-zero exit (1)" 


ip-172-31-25-121.ec2.internal 


ip-172-31-25-121.ec2.internal 


: non-zero exit (137)" 


ip-172-31-25-121.ec2.internal 


: non-zero exit (137)" 


ip-172-31-29-67.ec2.internal 


ip-172-31-29-67.ec2.internal 


: non-zero exit (1)" 


ip-172-31-2-177.ec2.internal 


: non-zero exit (1)" 
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While the rolling update is in progress, some of the running tasks could be based on the previous 
service specification (mysql:5.6), while others are based on the new service specification (mysql: latest). 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

vqc6rhzw5uxz mysql.1 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 4 seconds ago 

tvxjmahyo8uh mysql.2 mysql:5.6 ip-172-31-29-67.ec2.internal 
Running Running 11 seconds ago 

u2ea4xq4yx6t mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 13 seconds ago 

iw8vwsxq3tjz mysql.4 mysql:5.6 ip-172-31-25-121.ec2.internal 
Running Running 46 seconds ago 

f5vcf9mgluge mysql.5 mysql:5.6 ip-172-31-29-67.ec2.internal 
Running Running 23 seconds ago 


When the rolling update has completed, all running tasks are based on the new service specification. 


~ $ docker service ps -f desired-state-running mysql 
ID NAME IMAGE 
DESIRED STATE CURRENT STATE 


NODE 


ERROR PORTS 


vqc6rhzw5uxz mysql.1 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 45 seconds ago 

53choz0dd967 mysql.2 mysql: latest ip-172-31-29-67.ec2. internal 
Running Running less than a second ago 

u2ea4xq4yx6t mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 53 seconds ago 

tyo6vOyen7ev mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 21 seconds ago 

upt2120sx7au mysql.5 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 25 seconds ago 


Rolling Update to Add and Remove Environment Variables 


The Docker image mysql requires one mandatory environment variable MYSOL ROOT PASSWORD for the root 
password and supports some other environment variables that may also be specified. The other environment 
variables are MYSQL DATABASE for the MySQL database, MYSQL USER for the MYSQL user, MYSOL PASSWORD for the 
MySQL password, and MYSQL ALLOW EMPTY PASSWORD for whether to allow the root password to be empty. The 
MYSOL ROOT PASSWORD was already set when the mysql service was created. Using the - -env-add option to the 
docker service update command, we can add the other environment variables. 


~ $ docker service update --env-add MYSOL DATABASE-'mysqldb' --env-add MYSOL USER-'mysqgl' 
--env-add MYSQL PASSWORD-'mysql'  --env-add MYSQL ALLOW EMPTY PASSWORD-'no' --update- 
parallelism 1 mysql 


mysql 


An output of mysql implies the command ran successfully. 

The rolling update status is found with the docker service inspect command, which in addition to 
listing the env variables added in the Env JSON object, lists the UpdateStatus. The State of the update status 
is updating and the message is “update in progress”. 
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~ $ docker service inspect mysql 


[ 
oe 
"Spec": { 
"Name": "mysql", 
"ContainerSpec": { 
"Env": [ 
"MYSQL ROOT PASSWORD-mysql", 
"MYSQL DATABASE-mysqldb", 
"MYSQL USER-mysql", 
"MYSQL PASSWORD-mysql", 
"MYSOL ALLOW EMPTY PASSWORD-no" 
J» 
J, 
"UpdateStatus": { 
"State": "updating", 
"StartedAt": "2017-07-25T19:18:11.44139778Z", 
"Message": "update in progress" 
j 
j 
] 


When the update has completed, the UpdateStatus state becomes "completed" and the Message 
becomes "update completed". 


~ $ docker service inspect mysql 


Js 
"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T19:18:11.44139778Z", 
"CompletedAt": "2017-07-25T19:20:37.912993431Z", 
"Message": "update completed" 


As indicated by the StartedAt and CompletedAt timestamp, the rolling update takes about two minutes. 
Listing only tasks with desired state of running indicates that one task has been running for 21 seconds and 
another task has been running for two minutes. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

3zhf94kklu6r mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 21 seconds ago 

ta16ch5kjlr9 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 2 minutes ago 

fc7uxvwvcmk3 mysql.3 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running about a minute ago 


163 


CHAPTER 9 ~ ROLLING UPDATES 


jir97p344kol mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running about a minute ago 

5rly53mcc8yq mysql.5 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 45 seconds ago 


The environment variables added may be removed with another docker service update command 
and the - -env-rm options for each environment variable to remove. Only the env variable name is to be 
specified in - -env- rm, not the env value. 


~ $ docker service update --env-rm MYSOL DATABASE --env-rm MYSOL USER --env-rm 
MYSQL PASSWORD  --env-rm MYSOL ALLOW EMPTY PASSWORD mysql 


mysql 


Another rolling update gets performed. All service tasks get shut down and new service tasks based 
on the new service specification are started. The service definition lists only the mandatory environment 
variable MYSQL_ROOT PASSWORD. 


~ $ docker service inspect mysql 


ee 


"Env": [ 
"MYSOL ROOT PASSWORD-mysql" 
iF 
) 


"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T19:20:57.968668604Z", 
"CompletedAt": "2017-07-25T19:22:59.18517919Z", 
"Message": "update completed" 


Rolling Update to Set CPU and Memory Limits and Reserve 


A rolling update may be used to set new resource limits and reserves. 


~ $ docker service update --reserve-cpu 1 --limit-cpu 2 --reserve-memory 256mb - 
-limit-memory 512mb mysql 
mysql 


New resource limits and reserves are configured, as listed in the service specification. The 
Previous Spec indicates that no Resources Limits and Reservations are configured to start with. 


~ $ docker service inspect mysql 


"Spec": { 
"Name": "mysql", 


"ContainerSpec": { 


Jy 
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"Resources": { 

"Limits": { 
"NanoCPUs": 2000000000, 
"MemoryBytes": 536870912 

) 

"Reservations": ( 
"NanoCPUs": 1000000000, 
"MemoryBytes": 268435456 


Js 
)» 


"PreviousSpec": { 


"Name": "mysql", 
"Resources": ( 
"Limits": {}, 
"Reservations": {} 
Js 
"UpdateStatus": { 
"State": “updating”, 
"StartedAt": "2017-07-251T19:23:44.004458295Z", 
"Message": "update in progress" 


ROLLING UPDATES 


Setting new resource limits and reserves are subject to node capacity limits. If requested resources 


exceed the node capacity the rolling update may continue to run and not get completed, with some tasks in 
the pending current state. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

5u7zifw15n7t mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about an hour ago 

2kgsb16c8m8u mysql.2 mysql:latest 

Running Pending about an hour ago 

muo8iu9qzqlh mysql.3 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running about an hour ago 

aakxr8dw5s15 mysql.4 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running about an hour ago 

26045639f20p mysql.5 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about an hour ago 


If some tasks are pending, adding resources to the Swarm could make the pending tasks run. We 


can update the CloudFormation stack to increase the number of worker nodes from 2 to 3, as shown in 
Figure 9-2. 
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mp CloudFormation ~ Stacks > Stack Detail » Update Stack 


Update Docker stack 


at Sa specify Details 
| Specify Details 
Options Specily parameter values. You can use or change the defaull parameter values, which are defined in the AWS CloudFomation template, Learn more 


REVIEW 


Stack name Docker 


Parameters 
Swarm Size 
Number of Swarm 1 v Number of Swarm manager nodes (1, 3, 5) 
managers? 
Number of Swarm worker 3 Number of worker nodes in the Swarm (0-1000 


lè 


nodes? 


Figure 9-2. Increasing the number of worker nodes in the Swarm 


Subsequently, the Swarm should list four nodes. 


~ $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
8ih6uvu8ugdemnovzkg6v7mzg ip-172-31-2-177.ec2.internal Ready Active 
e7viginOluuoikynjnl33v9pa ip-172-31-29-67.ec2.internal Ready Active 
ptm7e0p346zwypos7wnpcm72d * ip-172-31-25-121.ec2.internal Ready Active Leader 
t4d0aq9w2a6avjx94zgkwc557 + ip-172-31-42-198.ec2.internal Ready Active 


With increased resources in the Swarm, the pending tasks also start to run. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

5u7zifw15n7t mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about an hour ago 

2kgsb16c8m8u mysql.2 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 7 minutes ago 

muo8iu9qzqlh mysql.3 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running about an hour ago 

15j2drlcm75f mysql.4 mysql:latest ip-172-31-42-198.ec2.internal 
Running Running 4 seconds ago 

26045639f20p mysql.5 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about an hour ago 
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Rolling Update to a Different Image 


Rolling update may also be used to update to a completely different Docker image. As an example, perform 
a rolling update to the mysql service to use Docker image postgres instead of the mysql image it is using. 
Other options such as - -update-parallelism may also be set. 


~ $ docker service update --image postgres --update-parallelism 1 mysql 
mysql 


The mysql: latest image-based tasks start to get shut down and postgres image-based replacement 
tasks begin to get started one task at a time. The rolling update does not get completed immediately and 
listing the service tasks with the desired state as running lists some tasks based on the postgres: latest 
image, while other tasks are still using the mysql: latest image. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

9tzm5pa6pcyx mysql.1 postgres:latest ip-172-31-2-177.ec2.internal 
Running Running 39 seconds ago 

xj23fu5svv9d mysql.2 postgres:latest ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

muo8iu9qzqlh mysql.3 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running about an hour ago 

skzxi33c6060 mysql.4 postgres:latest ip-172-31-2-177.ec2.internal 
Running Running 13 seconds ago 

26045639f20p mysql.5 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about an hour ago 


One replica at a time, the mysql image-based replicas are shut down and postgres image-based 
replicas are started. After about two minutes, all tasks have updated to the postgres: latest image. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

9tzm5pa6pcyx mysql.1 postgres:latest ip-172-31-2-177.ec2.internal 
Running Running about a minute ago 

xj23fu5svv9d mysql.2 postgres:latest ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

kd9pk31vpof2 mysql.3 postgres:latest ip-172-31-42-198.ec2.internal 
Running Running 35 seconds ago 

skzxi33c6060 mysql.4 postgres:latest ip-172-31-2-177.ec2.internal 
Running Running 59 seconds ago 

umtitiuvt5gg mysql.5 postgres:latest ip-172-31-25-121.ec2.internal 
Running Running 8 seconds ago 


The service name continues to be the same and the replica names also include the mysql prefix. The 
mysql service definition ContainerSpec lists the image as postgres. Updating the image to postgres does 
not imply that all other service definition settings are updated for the new image. The postgres image 
does not use the MYSOL ROOT PASSWORD, but the environment variable continues to be in the service 
specification. 
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~ $ docker service inspect mysql 
[ 
"Spec": { 
"Name": "mysql", 
"ContainerSpec": { 
"Image": “postgres: latest@sha256:e92fe21f695d27be7050284229a1c8c63ac10d8 
8cba58d779c243566e125aa34 , 
"Env": [ 
"MYSQL ROOT PASSWORD-mysq]" 
|, 


"PreviousSpec": { 
"Name": "mysql", 
"ContainerSpec": { 
"Image": “mysql: latest@sha256:75c563c474f1adc149978011fedfe2e6670483d133 
b22b07ee32789b626f8de3", 
"Env": [ 
"MYSQL ROOT PASSWORD=mysql" 


Js 
"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T120:39:45.230997671Z", 
"CompletedAt": "2017-07-25T120:42:04.186537673Z", 
"Message": "update completed" 


The MYSQL ROOT PASSWORD environment variable may be removed with another update command. 


~ $ docker service update  --env-rm MYSOL ROOT PASSWORD mysql 
mysql 


Subsequently, the ContainerSpec does not include the MYSOL ROOT PASSWORD environment variable. 


~ $ docker service inspect mysql 


"Spec": { 
"Name": “mysql”, 


"ContainerSpec": { 
"Image": “postgres: latest@sha256:e92fe21f695d27be7050284229a1c8c63ac10d8 
8cba58d779c243566e125aa34 , 
"StopGracePeriod": 10000000000, 
"DNSConfig": {} 


)» 
Jy 
"PreviousSpec": { 
"ContainerSpec": { 
"Image": “postgres: latest@sha256:e92fe21f695d27be7050284229a1c8c63ac10d8 


8cba58d779c243566e125aa34", 
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"Env": [ 


"MYSOL ROOT PASSWORD-mysql" 


J» 


"UpdateStatus": { 
"State": "updating", 
"StartedAt": "2017-07-25T20:42:56.651025816Z", 
"Message": "update in progress" 
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A rolling update to remove an environment variable involves shutting down all service tasks and 
starting all new tasks. The update takes about two minutes to complete. 


~ $ docker service inspect mysql 


J, 


"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T120:42:56.651025816Z", 
"CompletedAt": "2017-07-25T20:44:55.078906359Z", 
"Message": "update completed" 


Listing the running tasks indicates that tasks have only been running two minutes at the maximum. 


~ $ docker service ps -f desired-state-running mysql 


ID 

DESIRED STATE 
menpo2zgit5u 
Running 
adnid3t69sue 
Running 
we92apfuivil 
Running 
ed7vh4ozefm5 
Running 
12x2377ad7u0 
Running 


NAME IMAGE 
CURRENT STATE ERROR 
mysql.1 postgres 
Running about a minute ago 
mysql.2 postgres 
Running about a minute ago 
mysql.3 postgres 
Running 46 seconds ago 
mysql.4 postgres 
Running 2 minutes ago 
mysql.5 postgres 


Running about a minute ago 


: latest 


: latest 


: latest 


: latest 


: latest 


NODE 

PORTS 
ip-172-31-2-177.ec2.internal 
ip-172-31-25-121.ec2.internal 
ip-172-31-42-198.ec2.internal 


ip-172-31-29-67.ec2.internal 


ip-172-31-25-121.ec2.internal 


By removing the env variable MYSQL_ROOT PASSWORD the mysql service gets updated to use Docker image 
postgres. The service name itself cannot be updated. The service may be updated back to the mysql image 
and the mandatory environment variable MYSQL ROOT PASSWORD added with another rolling update. 


~ $ docker service update --image mysql --env-add MYSOL ROOT PASSWORD-'mysql' mysql 


mysql 
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Again, listing the replicas with a desired state as running lists the postgres image-based replicas being 
replaced by mysql image-based replicas. One replica at a time, the postgres image-based replicas are 
replaced by mysql image-based replicas. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

menpo2zgit5u mysql.1 postgres:latest ip-172-31-2-177.ec2.internal 

Running Running 2 minutes ago 

adnid3t69sue mysql.2 postgres: latest ip-172-31-25-121.ec2.internal 
Running Running 2 minutes ago 

we92apfuivil mysql.3 postgres:latest ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

pjvj50j822xr mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 

Running Running 12 seconds ago 

12x2377ad7u0 mysql.5 postgres:latest ip-172-31-25-121.ec2.internal 
Running Running 2 minutes ago 


Within a minute or two, all the postgres image replicas are replaced by mysql image-based replicas. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

sobd90v7gbmz mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about a minute ago 

st5t7y8rdgg1 mysql.2 mysql:latest ip-172-31-29-67.ec2.internal 

Running Running 57 seconds ago 

upekevrlbmgo mysql.3 mysql:latest ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

pjvj50j822xr mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 

Running Running 2 minutes ago 

nmrmdug87cyO mysql.5 mysql:latest ip-172-31-2-177.ec2.internal 

Running Running 2 minutes ago 


The service specification is updated to the mysql image and the mandatory environment variable 
MYSQL ROOT PASSWORD is added. When the update has completed, the UpdateStatus State becomes completed. 


~ $ docker service inspect mysql 
l 
"Spec": { 
"Name": "mysql", 


"Image": "mysql:latest@sha256:75c563c474f1adc149978011fedfe2e6670483d133 
b22b07ee32789b626f8de3", 
"Env": [ 
"MYSQL_ROOT_PASSWORD=mysq1" 
J» 


Jy 


"PreviousSpec": { 
"Name": "mysql", 
"ContainerSpec": { 
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"Image": “postgres: latest@sha256:e92fe21f695d27be7050284229a1c8c63ac10d8 
8cba58d779c243566e125aa34 , 
)» 
"UpdateStatus": { 

"State": "completed", 

"StartedAt": "2017-07-25T20:45:54.104241339Z", 

"CompletedAt": "2017-07-25T20:47:47.996420791Z", 

"Message": "update completed" 


Rolling Restart 


Docker 1.13 added a new option to perform a rolling restart even when no update is required based on the 
update options. As an example starting with the mysql service with update config as - -update-parallelism 
1 and - -update-delay 20s, the following update command won't perform any rolling update, as no changes 
are being made to the service. 


~ $ docker service update --update-parallelism 1 --update-delay 20s mysql 
mysql 


To force a rolling restart, include the - - force option. 


~ $ docker service update --force --update-parallelism 1 --update-delay 20s mysql 
mysql 


Service tasks begin to get shut down and new service tasks are started even though no update is made 
to the service specification. Some tasks are listed as having started a few seconds ago. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 
sobd90v7gbmz mysql.1 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 3 minutes ago 

trye9chir91l mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running 23 seconds ago 

uu7sfp147xnu mysql.3 mysql:latest ip-172-31-42-198.ec2.internal 
Running Running less than a second ago 

pjvj50j822xr mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 4 minutes ago 

nmrmdug87cyo mysql.5 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 3 minutes ago 


A rolling restart could take 1-2 minutes to complete. 


~ $ docker service inspect mysql 


[ 
E. 
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"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T20:49:34.716535081Z", 
"CompletedAt": "2017-07-25T120:51:36.880045931Z", 
"Message": "update completed" 


After the rolling restart has completed, the service has all new service tasks as shown. 


~ $ docker service ps -f desired-state-running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

z2n2qcgfsbke mysql.1 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 6 seconds ago 

trye9chir91l mysql.2 mysql:latest ip-172-31-25-121.ec2.internal 
Running Running about a minute ago 

uu7sfp147xnu mysql.3 mysql:latest ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

1aovurxkteq1 mysql.4 mysql:latest ip-172-31-29-67.ec2.internal 
Running Running 29 seconds ago 

rolslq6jibvp mysql.5 mysql:latest ip-172-31-2-177.ec2.internal 
Running Running 52 seconds ago 


Rolling Update to Add and Remove Mounts 


Rolling update can also be used to add and remove mounts. As an example, we add a mount of type volume 
with the source volume specified with src and the destination directory specified with dst. 


~ $ docker service update ^ 


> --mount-add type-volume,src-mysql-scripts,dst-/etc/mysql/scripts ^ 
> mysql 
mysql 


A mount is added to the service and is listed in the service definition. Adding a mount involves shutting 
down all service tasks and starting new tasks. The rolling update could take 1-2 minutes. 


~ $ docker service inspect mysql 


[ 
"Spec": { 
"ContainerSpec": { 


"Mounts": [ 


{ 
"Type": "volume", 
"Source": "mysql-scripts", 
"Target": "/etc/mysql/scripts" 
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"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T120:51:55.205456644Z", 
"CompletedAt": "2017-07-25T120:53:56.4513138267", 
"Message": "update completed" 


The mount added may be removed with the - -mount-rm option of the docker service update 
command and by supplying only the mount destination directory as an argument. 


~ $ docker service update \ 


> --mount-rm /etc/mysql/scripts \ 
> mysql 
mysql 


Another rolling update is performed and the mount is removed. It does not get listed in the service 
definition. The PreviousSpec lists the mount. The UpdateStatus indicates the status of the rolling update. 


~ $ docker service inspect mysql 
"Spec": { 
"Name": "mysql", 
"ContainerSpec": { 


"PreviousSpec": { 
"Name": "mysql", 


"Mounts": [ 
{ 
"Type": "volume", 
"Source": "mysql-scripts", 
"/etc/mysql/scripts" 


"Target": 
j 
"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T20:55:56. 308443242", 
"CompletedAt": "2017-07-25T20:57:58.489349432Z", 
"Message": "update completed" 


Rolling Update Failure Action 


The --update-failure-action option ofthe docker service create and docker service update 
commands specifies the follow-up action to take if the update to a task fails and returns FAILED. We set the 
UpdateConfig for the mysql service to include a --update-failure-action of pause (the default). The other 
option setting is continue, which does not pause a rolling update but continues with the update of the next 
task. To demonstrate a update failure action, specify a Docker image that does not exist, such as mysq1:5.9. 


173 


CHAPTER 9 ~ ROLLING UPDATES 


~ $ docker service update ^ 

> --replicas 10 \ 

>  --image mysql:5.9 \ 

> --update-delay 10s \ 

>  --update-failure-action pause \ 

> mysql 

image mysql:5.9 could not be accessed on a registry to record 
its digest. Each node will access mysql:5.9 independently, 
possibly leading to different nodes running different 
versions of the image. 


mysql 


The rolling update is still started and the update status indicates that the update is paused. The update 
status message indicates “update paused due to failure or early termination of task” 


~ $ docker service inspect mysql 
[ 
"Spec": { 
"Name": "mysql", 


32 

"UpdateConfig": { 
"Parallelism": 1, 
"Delay": 10000000000, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": 0, 
"Order": "stop-first" 

Js 

"RollbackConfig": { 
"Parallelism": 1, 
"FailureAction": "pause", 
"Monitor": 5000000000, 
"MaxFailureRatio": O, 
"Order": "stop-first" 

Js 


)» 
"UpdateStatus": { 


"State": “paused”, 

"StartedAt": "2017-07-25T20:58:51.695333064Z", 

"Message": "update paused due to failure or early termination of task 
s1p1nOx3k67uwpoj7qxg13747" 


Two options are available if a rolling update is paused due to update to a task having failed. 
e Restart a paused update using docker service update <SERVICE-ID>. 


e Ifan update failure is repeated, find the cause of the failure and reconfigure the 
service by supplying other options to the docker service update <SERVICE-ID> 
command. 
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Roll Back to Previous Specification 


Docker 1.13 Swarm mode added the feature to roll back to the previous service definition. As an example, 
perform a rolling update to update the image of the mysql service to postgres. The mysql-based replicas 
begin to be shut down and postgres-based replicas are started. At any time during the rolling update from 
the mysql image to the postgres image or after the update to the postgres image has completed, if it is 
ascertained that the rolling update should not have been started or performed, the rolling update may be 
rolled back with the following command. To demonstrate a rollback, we first start a mysql service. 


~ $ docker service rm mysql 


mysql 


~ $ docker service create \ 
>  --env MYSQL ROOT PASSWORD='mysql'\ 


--replicas 5 \ 
--name mysql \ 


--update-parallelism 1 \ 


> 
> 
> --update-delay 10s \ 
> 
> 


mysql:5.6 


xkmrhnkoa444zambp9yhimkgh 
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We start a rolling update to the postgres image from the mysql image. 


~ $ docker service update --image postgres 


mysql 


mysql 


Subsequently, some of the tasks are based on the postgres image and some on the mysql image. 


~ $ docker service ps mysql 


ID 

DESIRED STATE 
mnm5pg9ha61u 
Running 
9yOfzn4sgivO 
Ready 
ewl7zxwi07gc 
Shutdown 
13ock28cmtzx 
Running 
1vqs3lcqvbt5 
Running 
wu11jjbszesy 
Shutdown 
g3tr6z9l5vzx 
Running 


NAME IMAGE 
CURRENT STATE ERROR 
mysql.1 mysql:5.6 
Running 58 seconds ago 
mysql.2 postgres:latest 
Ready 2 seconds ago 

V mysql.2 mysql:5.6 
Running 2 seconds ago 
mysql.3 mysql:5.6 
Running 22 seconds ago 
mysql.4 postgres:latest 
Running 12 seconds ago 

V mysql.4 mysql:5.6 
Shutdown 13 seconds ago 
mysql.5 mysql:5.6 


Running 22 seconds ago 


Start a rollback to revert to the mysql image. 


~ $ docker service update --rollback mysql 


mysql 


NODE 

PORTS 
ip-172-31-25-121.ec2.internal 
ip-172-31-2-177.ec2. internal 
ip-172-31-2-177.ec2. internal 
ip-172-31-42-198.ec2.internal 
ip-172-31-29-67.ec2. internal 


ip-172-31-29-67.ec2. internal 


ip-172-31-42-198.ec2.internal 
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The postgres image-based tasks start to get shut down and the mysql image-based tasks are started. 


~ $ docker service ps mysql 


ID 


DESIRED STATE 


mnm5pg9ha61u 
Running 
gyqgtoc4ix3y 
Running 
9yOfzn4sgivO 
Shutdown 
ewl7zxwi07gc 
Shutdown 
130ck28cmtzx 
Running 
ecvh8fd5308k 
Running 
1vqs3lcqvbt5 
Shutdown 
wu11jjbszesy 
Shutdown 
m27d3gz4g6dy 
Running 
g3tr6z9l5vzx 
Shutdown 


NAME IMAGE 

CURRENT STATE ERROR 
mysql.1 mysql:5.6 
Running about a minute ago 

mysql.2 mysql:5.6 
Running 14 seconds ago 

V mysql.2 postgres:latest 
Shutdown 15 seconds ago 

V mysql.2 mysql:5.6 
Shutdown 23 seconds ago 
mysql.3 mysql:5.6 
Running 46 seconds ago 
mysql.4 mysql:5.6 
Running 16 seconds ago 

V mysql.4 postgres:latest 
Shutdown 16 seconds ago 

V mysql.4 mysql:5.6 
Shutdown 37 seconds ago 
mysql.5 mysql:5.6 
Running 1 second ago 

V mysql.5 mysql:5.6 


Failed 6 seconds ago 


NODE 
PORTS 

ip-172-31-25-121.ec2.internal 
ip-172-31-2-177.ec2.internal 
ip-172-31-2-177.ec2.internal 
ip-172-31-2-177.ec2.internal 
ip-172-31-42-198.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-29-67.ec2.internal 
ip-172-31-29-67.ec2.internal 


ip-172-31-25-121.ec2.internal 


ip-172-31-42-198.ec2.internal 


"task: non-zero exit (1)" 


The rolling update from mysql to postgres is rolled back. When the rollback has completed, all replicas 
are mysql image-based, which is the desired state of the service to start with. 


~ $ docker service ps -f desired-state-running mysql 


ID 


DESIRED STATE 


xamxi290kj74 
Running 
gyqgtoc4ix3y 
Running 
13ock28cmtzx 
Running 
ecvh8fd5308k 
Running 


NAME IMAGE 

CURRENT STATE ERROR 
mysql.1 mysql:5.6 
Running 30 seconds ago 

mysql.2 mysql:5.6 
Running 56 seconds ago 

mysql.3 mysql:5.6 
Running about a minute ago 

mysql.4 mysql:5.6 


Running 58 seconds ago 


Rolling Update on a Global Service 


A rolling update may also be performed on a global service. To demonstrate, we create a global service for 
the mysql: latest image. 


~ $ docker service rm mysql 


mysql 


~ ¢ docker service create \ 


> --mode global \ 


>  --env MYSQL ROOT PASSWORD='mysql'\ 


176 


NODE 

PORTS 
ip-172-31-25-121.ec2. internal 
ip-172-31-2-177.ec2. internal 


ip-172-31-42-198.ec2.internal 


ip-172-31-29-67.ec2. internal 
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»  --name mysql \ 
» mysql 
7nokncnti3izudO8gfdovwxwa 


Start a rolling update to Docker image mysql:5.6. ~ $ docker service update ^ 
>  --image mysql:5.6 \ 

>  --update-delay 10s \ 

» mysql 

mysql 


The service is updated. The Spec>ContainerSpec> Image is updated to mysql :5.6 from the PreviousSpec> 
ContainerSpec>Image of mysql: latest. 


~ $ docker service inspect mysql 
[ 
"Spec": { 
"Name": "mysql", 
"ContainerSpec": { 
"Image": "mysql:5.68sha256:6ad5bd392c9190f392e65fd21f6debc8b2a76fc54f139 
49f9b5bc6a0096a5285" , 


32 
"PreviousSpec": { 
"Name": "mysql", 
"ContainerSpec": { 
"Image": "mysql:latest@sha256:75c563c474f1adc149978011fedfe2e6670483d133 
b22b07ee32789b626f8de3", 
"UpdateStatus": { 
"State": "completed", 
"StartedAt": "2017-07-25T21:06:46.973666693Z", 
"CompletedAt": "2017-07-25T21:07:46.656023733Z", 
"Message": "update completed" 


Within a minute, all the new service tasks based on mysql :5.6 are started. 


~ $ docker service ps -f desired-state=running mysql 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

ybf4xpofte8l mysql.81h6uvu8uqoemnovzkg6v7mzg mysql:5.6 ip-172-31-2-177.ec2.internal 
Running Running 46 seconds ago 

7nq99jeil9no mysql.t4dOaq9w2a6avjx94zgkwc557 | mysql:5.6 ip-172-31-42-198.ec2.internal 
Running Running about a minute ago 

wcng24mq7e8m mysql.e7viginOluuoikynjnl33v9pa mysql:5.6 ip-172-31-29-67.ec2.internal 
Running Running about a minute ago 

qi4t2pyhra3w mysql. ptm7e0p346zwypos7wnpcm72d mysql:5.6 ip-172-31-25-121.ec2. internal 
Running Running about a minute ago 
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A rolling update cannot be performed on a global service to set replicas with the - - replicas option, as 
indicated by the message in the following docker service update command. 


~ $ docker service update ^ 

> | --image mysql \ 

>  --replicas 1 \ 

» mysql 

replicas can only be used with replicated mode 


As the output indicates, while replicas are set on a replicated service mysql, replicas are not set on the 
global service. 


Summary 


This chapter discussed rolling updates on a service. A rolling update on a service involves shutting down 
previous service tasks and updating the service definition to start new tasks. In the next chapter, we discuss 
configuring networking in Swarm mode. 
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CHAPTER 10 


Networking 





Networking on a Docker Engine is provided by a bridge network, the dockero bridge. The dockero bridge is local in 
scope to a Docker host and is installed by default when Docker is installed. All Docker containers run on a Docker 
host and are connected to the dockero bridge network. They communicate with each other over the network. 


The Problem 


The default dockero bridge network has the following limitations: 


e The bridge network is limited in scope to the local Docker host to provide container- 
to-container networking and not for multi-host networking. 


e The bridge network isolates the Docker containers on the host from external access. 
A Docker container may expose a port or multiple ports and the ports may be published 
on the host for an external client host access, as illustrated in Figure 10-1, but by default 
the dockero bridge does not provide any external client access outside the network. 


Exposed 
port 
published 
on the 
host 
interface 










ort Docker 
P Container 
Figure 10-1. The default dockerO bridge network 
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The Solution 


The Swarm mode (Docker Engine >=1.12) creates an overlay network called ingress for the nodes in the 
Swarm. The ingress overlay network is a multi-host network to route ingress traffic to the Swarm; external 
clients use it to access Swarm services. Services are added to the ingress network if they publish a port. 

The ingress overlay network has a default gateway and a subnet and all services in the ingress network 
are exposed on all nodes in the Swarm, whether a service has a task scheduled on each node or not. In 
addition to the ingress network, custom overlay networks may be created using the overlay driver. Custom 
overlay networks provide network connectivity between the Docker daemons in the Swarm and are used for 
service-to-service communication. Ingress is a special type of overlay network and is not for network traffic 
between services or tasks. Swarm mode networking is illustrated in Figure 10-2. 
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Figure 10-2. The Swarm overlay networks 


The following Docker networks are used or could be used in Swarm mode. 


The Ingress Network 


The ingress network is created automatically when Swarm mode is initialized. On Docker for AWS, the 
ingress network is available out-of-the-box because the managed service has the Swarm mode enabled by 
default. The default overlay network called ingress extends to all nodes in the Swarm, whether the node has 
a service task scheduled or not. The ingress provides load balancing among a service's tasks. All services 
that publish a port are added to the ingress network. Even a service created in an internal network is added 
to ingress ifthe service publishes a port. If a service does not publish a port, itis not added to the ingress 
network. A service publishes a port with the - -publish or -p option using the following docker service 
create command syntax. 
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docker service create \ 
--name «SERVICE-NAME» \ 
--publish <PUBLISHED-PORT>:<TARGET-PORT> \ 
«IMAGE» 


If the <PUBLISHED-PORT> is omitted, the Swarm manager selects a port in the range 30000-32767 to 
publish the service. 
The following ports must be open between the Swarm nodes to use the ingress network. 


e Port 7946 TCP/UDP is used for the container network discovery 


e Port 4789 UDP is used for the container ingress network 


Custom Overlay Networks 


Custom overlay networks are created using the overlay driver and services may be created in the overlay 
networks. A service is created in an overlay network using the - -network option of the docker service 
create command. Overlay networks provide service-to-service communication. One Docker container 

in the overlay network can communicate directly with another Docker container in the network, whether 
the container is on the same node or a different node. Only Docker containers for Swarm service tasks can 
connect with each using the overlay network and not just any Docker containers running on the hosts in a 
Swarm. Docker containers started with the docker run «img» command, for instance, cannot connect to 
a Swarm overlay network, using docker network connect «overlay network» «container» for instance. 
Nor are Docker containers on Docker hosts that are not in a Swarm able to connect and communicate with 
Docker containers in the Swarm directly. Docker containers in different Swarm overlay networks cannot 
communicate with each other directly, as each Swarm overlay network is isolated from other networks. 

While the default overlay network in a Swarm, ingress, extends to all nodes in the Swarm whether a 
service task is running on it or not, a custom overlay network whose scope is also the Swarm does not extend 
to all nodes in the Swarm by default. A custom Swarm overlay network extends to only those nodes in the 
Swarm on which a service task created with the custom Swarm overlay network is running. 

An "overlay" network overlays the underlay network of the hosts and the scope of the overlay network is 
the Swarm. Service containers in an overlay network have different IP addresses and each overlay network 
has a different range of IP addresses assigned. On modern kernels, the overlay networks are allowed to 
overlap with the underlay network, and as a result, multiple networks can have the same IP addresses. 


The docker gwbridge Network 


Another network that is created automatically (in addition to the ingress network) when the Swarm mode is 
initialized is the docker gwbridge network. The docker gwbridge network is a bridge network that connects 
all the overlay networks, including the ingress network, to a Docker daemon's host network. Each service 
container is connected to the local Docker daemon host's docker gwbridge network. 


The Bridge Network 


A bridge network is a network on a host that is managed by Docker. Docker containers on the host 
communicate with each other over the bridge network. A Swarm mode service that does not publish a port 
is also created in the bridge network. So are the Docker containers started with the docker run command. 
This implies that a Swarm mode Docker service that does not publish a port is in the same network as 
Docker containers started with the docker run command. 
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This chapter covers the following topics: 
e Setting the environment 
e Networking in Swarm mode 
e Using the default overlay network ingress to create a service 
e Creating a custom overlay network 
e Using a custom overlay network to create a service 
e Connecting to another Docker container in the same overlay network 
e Creating an internal network 


e Deleting a network 


Setting the Environment 


Create a three-node Docker Swarm on Docker for AWS, as discussed in Chapter 3. An AWS CloudFormation 
stack, shown in Figure 10-3, is used to create a Swarm. 


(p CloudFormation v Stacks 





eo Introducing StackSets 


AW tack $a container for a set of AWS uU, ormanhon stacks anc allows to create stacks across qf)ple AVVS Accounts and ANS Regions en the Stat kSets console to oet 


Stack Name Created Time Status Description 


REATE MPLETE ker CE for AWS 17 Of eii7 Of @-aws2 


Figure 10-3. AWS CloudFormation stack 
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Obtain the public IP address of the Swarm manager node, as shown in Figure 10-4. 


Launch Instance Connect Actions ¥ 
4 


o 9 9 
Q Filter by tags and attributes or search by keyv o 1t030f3 
Name Instance ID Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 
@ Docker-Manager i-083eb8f9739e82ee1 — t2 micro us-east-1c Q running © 2/2 checks None ‘je ee2-174-129-48-148.co 
Docker-worker i-Od260eb7bSb 1b49 Q2 micro us-east-1c @ running Q 2/2 checks None Me  e¢2-54-209-159-170.c0 
Docker-worker i-O9717d6de4f309b4e —— Q.micro us-east-1a @ running Q 2/2checks None YW — 09c2-34-205-41-131.co 


Instance: | i-082eb8f9739e82ee1 (Docker-Manager) 


Description Status Checks 


Instance ID 


Instance state 
Instance type 
Elastic IPs 
Availability zone 


security groups 


Scheduled events 


AMI ID 


Monitoring Tags 


-083eb919/ 39682661 


Public DNS (IPv4) 


running no IPv4 Public IP 
t2. micro * IPv6 IPs 

Prvate DNS 
us-east-1c Private IPs 


Docker-ManagerVpc SG-1C8V48XMF IHA3 
Docker-SwarmWideSG- TUWOUXM4BNBGT 
view inbound rules 


Secondary private IPs 


No scheduled events VPC ID 


Moby Linux 17 06 O-ce-aws2 stable (ami- Subnet ID 


Public ONS: ec2-174-129-48-148.compute-1.amazonaws.com 


mmpms:- 


ec2-174-129-45-148. compute- 
1.amazonaws com 


174.129 48.148 


ip-172-31-47-15 ec2 intemal 


172.31.47.15 


vpc-b35b30ca 


subnet-c7c59c8f X 


Figure 10-4. Obtaining the public IP address of a Swarm manager node instance 


SSH login into the Swarm manager instance. 


[root@localhost ~]# ssh -i "docker.pem" docker@174.129.48.148 


Welcome to Docker! 


List the Swarm nodes—one manager and two worker nodes. 


~ $ docker node ls 


ID HOSTNAME STATUS 
npz2akark8etv4ib9biob5yyk ip-172-31-47-123.ec2.internal Ready 
p6wat4lxq6aio3h4fp2ikgwor ip-172-31-3-168.ec2.internal Ready 
tb5agvzbidrupq7b83tk00cx3 * ip-172-31-47-15.ec2.internal Ready 


Networking in Swarm Mode 


The Swarm mode provides some default networks, which may be listed with the docker network ls 
command. These networks are available not just on Docker for AWS but on any platform (such as CoreOS) 


in Swarm mode. 


~ $ docker network ls 


AVAILABILITY MANAGER STATUS 
Active 
Active 


Active Leader 


NETWORK ID NAME DRIVER SCOPE 
34a5f77de8cf bridge bridge local 
0e06b811a613 docker gwbridge bridge local 
6763ebad69cf host host local 
e41an60iwval ingress overlay swarm 
eb7399d3ffdd none null local 
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We discussed most of these networks in a preceding section. The "host" network is the networking 
stack of the host. The "none" network provides no networking between a Docker container and the host 
networking stack and creates a container without network access. 

The default networks are available on a Swarm manager node and Swarm worker nodes even before 
any service task is scheduled. 

The listed networks may be filtered using the driver filter set to overlay. 


docker network ls --filter driver-overlay 
Only the ingress network is listed. No other overlay network is provisioned by default. 


~ $ docker network ls --filter driver-overlay 
NETWORK ID NAME DRIVER SCOPE 
e41an60iwval ingress overlay swarm 


The network of interest is the overlay network called ingress, but all the default networks are discussed 
in Table 10-1 in addition to being discussed in the chapter introduction. 


Table 10-1. Docker Networks 


Network Description 


bridge The bridge network is the dockerO network created on all Docker hosts. The 
Docker daemon connects containers to the dockerO network by default. Any Docker 
container started with the docker run command, even on a Swarm node, connects to 
the dockero bridge network. 


docker gwbridge Used for communication among Swarm nodes on different hosts. The network is 
used to provide external connectivity to a container that lacks an alternative network 
for connectivity to external networks and other Swarm nodes. When a container is 
connected to multiple networks, its external connectivity is provided via the first non- 
internal network, in lexical order. 


host Adds a container to the host's network stack. The network configuration inside the 
container is the same as the host's. 

ingress The overlay network used by the Swarm for ingress, which is external access. The 
ingress network is only for the routing mesh/ingress traffic. 

none Adds a container to a container specific network stack and the container lacks a 
network interface. 


The default networks cannot be removed and, other than the ingress network, a user does not need to 
connect directly or use the other networks. To find detailed information about the ingress network, run the 
following command. 


docker network inspect ingress 
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The ingress network's scope is the Swarm and the driver used is overlay. The subnet and gateway are 
10.255.0.0/16 and 10.255.0.1, respectively. The ingress network is not an internal network as indicated 
by the internal setting of false, which implies that the network is connected to external networks. The 
ingress network has an IPv4 address and the network is not IPv6 enabled. 


~ $ docker network inspect ingress 
[ 
{ 
"Name": “ingress”, 
"Id": "e41an60iwvalbeq5y3stdfem9", 
"Created": "2017-07-26118:38:29.753424199Z", 
"Scope": "swarm", 
"Driver": "overlay", 
"EnableIPv6": false, 
"IPAM": { 
"Driver": "default", 
"Options": null, 
"Config": [ 
{ 
"Subnet": "10.255.0.0/16", 
"Gateway": "10.255.0.1" 


] 
J, 


"Internal": false, 

"Attachable": false, 

"Ingress": true, 

"ConfigFrom": { 
"Network": "" 

Jo 


"ConfigOnly": false, 
"Containers": { 
"ingress-sbox": { 
"Name": "ingress-endpoint", 
"EndpointID": "f646b5cc4316994b8f9e50412e7c82550bc7ce733db70df 3f 
66b8d771d0f53c4", 

"MacAddress": "02:42:0a:ff:00:02", 
"IPv4Address": "10.255.0.2/16", 


"IPv6Address": "" 
j 
Jy 
"Options": { 
"com.docker.network.driver.overlay.vxlanid list": "4096" 
Jy 
"Labels": {}, 
"Peers": [ 
{ 
"Name": "ip-172-31-47-15.ec2.internal-17c7f752fb1a", 
"IP": "172.31.47.15" 
)» 
{ 
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"Name": "ip-172-31-47-123.ec2.internal-d6ebe8111adf", 
"IP": "172.31.47.123" 


)» 

{ 
"Name": "ip-172-31-3-168.ec2.internal-99510f4855ce", 
"IP": "172.31.3.168" 

j 


Using the Default Bridge Network to Create a Service 


To create a service in Swarm mode using the default bridge network, no special option needs to be specified. 
The - -publish or -p option must not be specified. Create a service for the mysql database. 


~ $ docker service create \ 
--env MYSQL ROOT PASSWORD- 'mysql'N 
--replicas 1 \ 
--name mysql ^ 
mysql 
ikujs72e46tisgoixjtksnky 


Mm v v v Vv 


The service is created and the service task is scheduled on one of the nodes. 


" $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
likujs72e46t mysql replicated 1/1 mysql:latest 


The service may be scaled to run tasks across the Swarm. 
~ $ docker service scale mysql-3 


mysql scaled to 3 
~ $ docker service ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

v4bn24seygc6 mysql.1 mysql:latest ip-172-31-47-15.ec2.internal 

Running Running 2 minutes ago 

29702ebj52gs mysql.2 mysql:latest ip-172-31-47-123.ec2.internal 
Running Running 3 seconds ago 

c7b8v16msudl mysql.3 mysql:latest ip-172-31-3-168.ec2.internal 

Running Running 3 seconds ago 


The mysql service created is not added to the ingress network, as it does not publish a port. 
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Creating a Service in the Ingress Network 


In this section, we create a Docker service in the ingress network. The ingress network is not to be 
specified using the - -network option of docker service create. A service must publish a port to be created 
in the ingress network. Create a Hello World service published (exposed) on port 8080. 


~ $ docker service rm hello-world 
hello-world 
~ $ docker service create \ 


>  --name hello-world \ 
> -p 8080:80\ 
>  --replicas 3 \ 


>  tutum/hello-world 
176ukzrctq22mn97dmgooatup 


The service creates three tasks, one on each node in the Swarm. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
l76ukzrctq22 hello-world replicated 3/3 tutum/hello-world:latest *:8080->80/tcp 
~ $ docker service ps hello-world 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 


5ownzdjdtiyu hello-world.1  tutum/hello-world: latest ip-172-31-14-234.ec2.internal 
Running Running 33 seconds ago 

csgofrbrznhq hello-world.2 | tutum/hello-world:latest ip-172-31-47-203.ec2.internal 
Running Running 33 seconds ago 

sctlt9rvn571  hello-world.3 | tutum/hello-world:latest ip-172-31-35-44.ec2.internal 
Running Running 32 seconds ago 
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The service may be accessed on any node instance in the Swarm on port 8080 using the «Public DNS»: 
«8080» URL. If an elastic load balancer is created, as for Docker for AWS, the service may be accessed at 
«LoadBalancer DNS>:<8080>, as shown in Figure 10-5. 


[^ Hello world! x 


Š Q | @ docker-externalloa-gtt5yh5as8wo-331359979.us-east-1.elb.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is b65723d5a72c 





Figure 10-5. Invoking a Docker service in the ingress network using EC2 elastic load balancer public DNS 


The <PublishedPort> 8080 may be omitted in the docker service create command. 


~ $ docker service create \ 
--name hello-world \ 

-p 80\ 

--replicas 3 \ 
tutum/hello-world 
pbjcjhx163wm37d5cc5au2fog 


? 
> 
> 
> 


Three service tasks are started across the Swarm. 
~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
pbjcjhx163wm hello-world replicated 3/3 tutum/hello-world:latest *:0->80/tcp 
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~ $ docker service ps hello-world 


ID NAME IMAGE 

DESIRED STATE CURRENT STATE ERROR PORTS 
xotbpvlo508n hello-world.1 . tutum/hello-world:latest 
Running Running 13 seconds ago 

nvdn3j5pzugi hello-world.2 . tutum/hello-world:latest 
Running Running 13 seconds ago 

uuveltc5izpl hello-world.3 | tutum/hello-world:latest 
Running Running 14 seconds ago 
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NODE 
ip-172-31-37-130.ec2.internal 
ip-172-31-44-205.ec2.internal 


ip-172-31-15-233.ec2.internal 


The Swarm manager automatically assigns a published port (30000), as listed in the docker service 


inspect command. 


~ $ docker service inspect hello-world 
l 
"Spec": { 
"Name": "hello-world", 


"EndpointSpec": { 
"Mode": "vip", 
"Ports": [ 
{ 
"Protocol": "tcp", 
"TargetPort": 80, 
"PublishMode": "ingress" 


j 
] 
j 
Js 
"Endpoint": { 
"Spec": { 
"Mode": "vip", 
"Ports": [ 
{ 
"Protocol": "tcp", 
"TargetPort": 80, 
"PublishMode": "ingress" 
j 
] 
Js 
"Ports": [ 
{ 
"Protocol": "tcp", 
"TargetPort": 80, 
"PublishedPort": 30000, 
"PublishMode": "ingress" 
j 
IF 
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"VirtualIPs": [ 


{ 
“NetworkID": "bllwwocjw5xejffmy6n8nhgm8", 


"Addr": "10.255.0.5/16" 


Even though the service publishes a port (30000 or other available port in the range 30000-32767), 
the AWS elastic load balancer for the Docker for AWS Swarm does not add a listener for the published 
port (30000 or other available port in the range 30000-32767). We add a listener with <Load Balancer 
Port:Instance Port> mapping of 30000:30000, as shown in Figure 10-6. 


Create Load Balancer Actions “ 
EM cveate Load Balancer oto 


Filter: Q Search X 1t50120f 1 
B Name - DNS name - State = MPCID - Availability Zones ~ Type 
Bü —Docker-ExtemnalLoa-LPHBAXKEV140 Docker-Extemalloa-LPHSA... vpc-ldc?b464 us-east-1a, us-east-Tb... classic 
Load balancer: | Docker-ExternalLoa-LPH5AXKEV 140 m m rm 
Description Instances Health Check Listeners Monitoring Tags 


The following listeners are currently configured for this load balancer 


Load Balancer Protocol Load Balancer Port Instance Protocol Instance Port Cipher X SSL Certificate 


TEP 30000 TCP 20000 WA NA 


^ 


Edit 


Figure 10-6. Adding a load balancer listener 
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Invoke the service at the <Load Balancer DNS>:<30000> URL, as shown in Figure 10-7. 


Hy [ Hello world! x 
r 


G docker-externalloa-Iph5axkev140-1316808033.us-east- 1.elb.amazonaws.com:3000C 


tutum 


Hello world! 


My hostname is 3895de571703 





Figure 10-7. Invoking a Hello World service on port 30000 


Creating a Custom Overlay Network 


We used the default overlay network ingress provisioned in Swarm mode. The ingress network is only for 
the Swarm mode routing mesh in which all nodes are included. The Swarm routing mesh is provided so 
that each node in the Swarm may accept connections on published ports for services in the Swarm even if a 
service does not run a task on a node. The ingress network is not for service-to-service communication. 

A custom overlay network may be used in Swarm mode for service-to-service communication. Next, 
create an overlay network using some advanced options, including setting subnets with the - -subnet option 
and the default gateway with the - -gateway option, as well as the IP range with the - -ip-range option. 

The - -driver option must be set to overlay and the network must be created in Swarm mode. A matching 
subnet for the specified IP range must be available. A subnet is a logical subdivision of an IP network. The 
gateway is a router that links a host's subnet to other networks. The following command must be run from a 
manager node. 


~ $ docker network create \ 
--subnet-192.168.0.0/16 \ 
--subnet=192.170.0.0/16 \ 
--gateway=192.168.0.100 \ 
--gateway=192.170.0.100 \ 
--ip-range-192.168.1.0/24 \ 


VV VV MV 
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> --driver overlay \ 
>  mysql-network 
mkileuo6ve329jx5xbdim6rio 


The custom overlay network is created and listed in networks as an overlay network with Swarm scope. 


~ $ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
34a5f77de8cf bridge bridge local 
0e06b811a613 docker_gwbridge bridge local 
6763ebad69cf host host local 
e41an60iwval ingress overlay swarm 
mkileuo6ve32 mysql-network overlay swarm 
eb7399d3ffdd none null local 


Listing only the overlay networks should list the ingress network and the custom mysql-network. 


~ $ docker network ls --filter driver-overlay 


NETWORK ID NAME DRIVER SCOPE 
e41an60iwval ingress overlay swarm 
mkileuo6ve32 mysql-network overlay swarm 


The detailed information about the custom overlay network mysql-network lists the subnets and 
gateways. 


~ $ docker network inspect mysql-network 
l 
{ 
"Name": "mysql-network", 
"Id": "mkileuo6ve329jx5xbd1m6r10", 
"Created": "0001-01-01T00:00:00Z", 
"Scope": "swarm", 
"Driver": "overlay", 
"EnableIPv6": false, 
"IPAM": { 
"Driver": "default", 
"Options": null, 
"Config": [ 
{ 
"Subnet": "192.168.0.0/16", 
"IPRange": "192.168.1.0/24", 
"Gateway": "192.168.0.100" 


)» 
{ 
"Subnet": "192.170.0.0/16", 
"Gateway": "192.170.0.100" 
j 


J, 


"Internal": false, 
"Attachable": false, 
"Ingress": false, 
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"ConfigFrom": { 
"Network": "" 
J, 


"ConfigOnly": false, 
"Containers": null, 
"Options": { 
"com.docker.network.driver.overlay.vxlanid list": "4097,4098" 
Jy 


"Labels": null 


Only a single overlay network can be created for specific subnets, gateways, and IP ranges. Using a 
different subnet, gateway, or IP range, a different overlay network may be created. 


~ $ docker network create ^ 
--subnet=10.0.0.0/16 \ 
--gateway=10.0.0.100 \ 
--ip-range-10.0.1.0/24 \ 
--driver overlay \ 
mysql-network-2 
quwgb1lwycgvogoq9t62ea4ny1 


VV VV MV 


The mysql-network-2 is created and added to the list of networks. 


~ $ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
34a5f77de8cf bridge bridge local 
0e06b811a613 docker_gwbridge bridge local 
6763ebad69cf host host local 
e41an60iwval ingress overlay swarm 
mkileuo6ve32 mysql-network overlay swarm 
qwgb1lwycgvo mysql-network-2 overlay Swarm 
eb7399d3ffdd none null local 


New overlay networks are only made available to worker nodes that have containers using the overlay. 
While the new overlay networks mysql-network and mysql-network-2 are available on the manager node, 
the network is not extended to the two worker nodes. SSH login to a worker node. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.209.159.170 
Welcome to Docker! 


The mysql-network and mysql-network-2 networks are not listed on the worker node. 


~ ¢ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
255542d86c1b bridge bridge local 
3a4436cOfboo docker gwbridge bridge local 
bddobe4885e9 host host local 
e41an60iwval ingress overlay swarm 
5C5f44ec3933 none null local 
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To extend the custom overlay network to worker nodes, create a service in the network that runs a task 
on the worker nodes, as we discuss in the next section. 

The Swarm mode overlay networking is secure by default. The gossip protocol is used to exchange 
overlay network information between Swarm nodes. The nodes encrypt and authenticate the information 
exchanged using the AES algorithm in GCM mode. Manager nodes rotate the encryption key for gossip data 
every 12 hours by default. Data exchanged between containers on different nodes on the overlay network 
may also be encrypted using the --opt encrypted option, which creates IPSEC tunnels between all the 
nodes on which tasks are scheduled. The IPSEC tunnels also use the AES algorithm in GCM mode and rotate 
the encryption key for gossip data every 12 hours. The following command creates an encrypted network. 


~ $ docker network create ^ 
>  --driver overlay \ 

>  --opt encrypted \ 

>  overlay-network-2 
aqppoe3qpy6mz1n46g5tunecr 


A Swarm scoped network that is encrypted is created. 


~ $ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
34a5f77de8cf bridge bridge local 
0e06b811a613 docker gwbridge bridge local 
6763ebad69cf host host local 
e41an60iwval ingress overlay swarm 
mkileuo6ve32 mysql-network overlay swarm 
qwgb1lwycgvo mysql-network-2 overlay Swarm 
eb7399d3ffdd none null local 
aqppoe3qpy6m overlay-network-2 overlay swarm 


Using a Custom Overlay Network to Create a Service 


If a custom overlay network is used to create a service, the - -network must be specified. The following 
command creates a MySQL database service in Swarm mode using the custom Swarm scoped overlay 
network mysql- network. 


~ $ docker service create \ 


> --env MYSOL ROOT PASSWORD-'mysql'* 
>  --replicas 1 \ 

>  --network mysql-network \ 

>  --name mysql-2\ 

> mysql 


ocd9sz8qqp2becfOww2rj5p5n 


The mysql-2 service is created. Scale the mysql-2 service to three replicas and lists the service tasks for 
the service. 


~ $ docker service scale mysql-2-3 
mysql-2 scaled to 3 
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Docker containers in two different networks for the two services—mysql (bridge network) and mysql-2 
(mysql-network overlay network)—are running simultaneously on the same node. 

A custom overlay network is not extended to all nodes in the Swarm until the nodes have service tasks 
that use the custom network. The mysql-network does not get extended to and get listed on a worker node 
until after a service task for mysql-2 has been scheduled on the node. 

A Docker container managed by the default Docker Engine bridge network dockerO cannot connect 
with a Docker container in a Swarm scoped overlay network. Using a Swarm overlay network in a docker 
run command, connecting with a Swarm overlay network with a docker network connect command, or 
linking a Docker container with a Swarm overlay network using the -- link option of the docker network 
connect command is not supported. The overlay networks in Swarm scope can only be used by a Docker 
service in the Swarm. 

For connecting between service containers: 


e Docker containers for the same or different services in the same Swarm scoped 
overlay network are able to connect with each other. 


e Docker containers for the same or different services in different Swarm scoped 
overlay networks are not able to connect with each other. 


In the next section, we discuss an internal network, but before we do so, the external network should be 
introduced. The Docker containers we have created as of yet are external network containers. The ingress 
network and the custom overlay network mysql-network are external networks. External networks provide 
a default route to the gateway. The host and the wider Internet network may connect to a Docker container 
in the ingress or custom overlay networks. As an example, run the following command to ping google.com 
from a Docker container's bash shell; the Docker container should be in the ingress overlay network or a 
custom Swarm overlay network. 


docker exec -it <containerid> ping -c 1 google.com 
A connection is established and data is exchanged. The command output is shown in italics. 


~ $ docker exec -it 3762d7c4ea68 ping -c 1 google.com 
PING google.com (172.217.7.142): 56 data bytes 

64 bytes from 172.217.7.142: icmp seq=0 ttl=47 time=0.703 ms 
--- google.com ping statistics --- 

1 packets transmitted, 1 packets received, 0% packet loss 
round-trip min/avg/max/stddev - 0.703/0.703/0.703/0.000 ms 


Creating an Internal Overlay Network 


In this section, we discuss creating and using an internal overlay network. An internal network does not 
provide external connectivity. What makes a network internal is that a default route to a gateway is not 
provided for external connectivity from the host or the wider Internet. 

First, create an internal overlay network using the - - internal option of the docker network create 
command. Add some other options, such as - - label, which have no bearing on the internal network. It's 
configured with the - - internal option ofthe docker network create command. 


" $ docker network create \ 
--subnet-10.0.0.0/16 \ 
--gateway-10.0.0.100 ^ 
--internal \ 

--label HelloWorldService \ 
--ip-range-10.0.1.0/24 \ 


VV VV ON 
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>  --driver overlay \ 
>  hello-world-network 
pfwsrjeakomplo5zm6t4p19a9 


The internal network is created and listed just the same as an external network would be. 


~ $ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
194d51d460e6 bridge bridge local 
a0674c5f1a4d docker_gwbridge bridge local 
pfwsrjeakomp hello-world-network overlay Swarm 
03a68475552f host host local 
tozyadpo6rxr ingress overlay swarm 
3dbd3c3ef439 none null local 


In the network description, the internal is set to true. 
core@ip-172-30-2-7 ~ $ docker network inspect hello-world-network 
[ 

{ 

"Name": "hello-world-network", 
"Id": "58fzvj4arudk2053q6k2t8rrk", 
"Scope": "swarm", 
"Driver": "overlay", 
"EnableIPv6": false, 
"IPAM": ( 
"Driver": "default", 
"Options": null, 
"Config": [ 


"Subnet": "10.0.0.0/16", 
"IPRange": °10.0.1.0/24", 
"Gateway": "10.0.0.100" 


] 
Jy 


"Internal": true, 

"Containers": null, 

"Options": { 
"com.docker.network.driver.overlay.vxlanid list": "257" 

Jy 


"Labels": { 
"HelloWorldService": 
} 


Create a service that uses the internal network with the - -network option. 
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" $ docker service create \ 

>  --name hello-world \ 

>  --network hello-world-network \ 
>  --replicas 3 \ 

>  tutum/hello-world 
hm5pfeftcvphdrd2zm3pp4lpj 


The service is created and the replicas are scheduled. 
Obtain the container ID for one of the service tasks, d365d4a5ff4c. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

d365d4a5ff4c | tutum/hello-world:latest ‘/bin/sh -c 'php-f..." | About a minute ago 
Up About a minute hello-world.3.r759ddnlidei1spoOzdi7xj4z 


As before, ping google.com from the Docker container. 
docker exec -it <containerid> ping -c 1 google.com 
A connection is not established, which is because the container is in an internal overlay network. 


~ $ docker exec -it d365d4a5ff4c ping -c 1 google.com 
ping: bad address 'google.com' 


Connection is established between containers in the same internal network, as the limitation is only on 
external connectivity. To demonstrate, obtain the container ID for another container in the same internal 


network. 


~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 
STATUS PORTS NAMES 

b7b505f5eb8d  tutum/hello-world:latest ^ "/bin/sh -c 'php-f..." 3 seconds ago 
Up 2 seconds hello-world.6.160ezt6da2t1odwdjvecb75fx 

57e612f35a38  tutum/hello-world:latest ^ "/bin/sh -c 'php-f..." 3 seconds ago 
Up 2 seconds hello-world.7.6ltqnybn8twhtblpqjtvulkup 

d365d4a5ff4c | tutum/hello-world:latest ^ "/bin/sh -c 'php-f..." 7 minutes ago 
Up 7 minutes hello-world.3.r759ddnlidei11spoOzdi7xj4z 


Connect between two containers in the same internal network. A connection is established. 


~ $ docker exec -it 4d365d4a5ff4c ping -c 1 57e612f35a38 
PING 57e612f35a38 (10.0.1.7): 56 data bytes 
64 bytes from 10.0.1.7: seq=0 ttl-64 time=0.288 ms 


--- 57e612135a338 ping statistics --- 


1 packets transmitted, 1 packets received, 0% packet loss 
round-trip min/avg/max - 0.288/0.288/0.288 ms 
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If a service created in an internal network publishes (exposes) a port, the service gets added to the 
ingress network and, even though the service is in an internal network, external connectivity is provisioned. 
As an example, we add the - -publish option ofthe docker service create command to publish the 
service on port 8080. 


" $ docker service create \ 

--name hello-world \ 

--network hello-world-network \ 
--publish 8080:80 \ 

--replicas 3 \ 

tutum/hello-world 
mqgek4umisgycagy4qa206f9c 


Vv VV VM 


Find a Docker container ID for a service task. 


~ $ docker ps 

CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

1c52804dc256 tutum/hello-world:latest ^ "/bin/sh -c 'php-f..." 28 seconds ago 
Up 27 seconds  80/tcp  hello-world.1.20152n01ng3t6uaiahpex9n4f 


Connect from the container in the internal network to the wider external network at google.com, as an 
example. A connection is established. Command output is shown in italics. 


~ $ docker exec -it 1c52804dc256 ping -c 1 google.com 
PING google.com (172.217.7.238): 56 data bytes 
64 bytes from 172.217.7.238: seq=0 ttl-47 time-1.076 ms 


--- google.com ping statistics --- 


1 packets transmitted, 1 packets received, 0% packet loss 
round-trip min/avg/max - 1.076/1.076/1.076 ms 


Deleting a Network 


A network that is not in use may be removed with the docker network rm «networkid» command. Multiple 
networks may be removed in the same command. As an example, we can list and remove multiple networks. 


~ $ docker network ls 


NETWORK ID NAME DRIVER SCOPE 
34a5f77de8cf bridge bridge local 
0e06b811a613 docker gwbridge bridge local 
wozpfgo8vbmh hello-world-network Swarm 
6763ebad69cf host host local 
e41an60iwval ingress overlay swarm 
mkileuo6ve32 mysql-network overlay swarm 
qwgbilwycgvo mysql-network-2 overlay swarm 
eb7399d3ffdd none null local 
aqppoe3qpyóm overlay-network-2 overlay swarm 
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Networks that are being used by a service are not removed. The command output is shown in italics. 


~ $ docker network rm hello-world-network mkileuo6ve32 qwgbilwycgvo overlay-network-2 
hello-world-network 

Error response from daemon: rpc error: code = 9 desc = network mkileuo6ve329jx5xbdim6r1o is 
in use by service ocd9sz8qqp2becfOww2rj5ps5nqwgb1lwycgvo 

overlay-network-2 


Summary 


This chapter discussed the networking used by the Docker Swarm mode. The default networking used in 
Swarm mode is the overlay network ingress, which is a multi-host network spanning all Docker nodes 

in the same Swarm to provide a routing mesh for each node to be able to accept ingress connections 

for services on published ports. Custom overlay network may be used to create a Docker service with the 
difference that a custom overlay network provides service-to-service communication instead of ingress 
communication and extends to a Swarm worker node only if a service task using the network is scheduled 
on the node. The chapter also discussed the difference between an internal and an external network. In the 
next chapter, we discuss logging and monitoring in Docker Swarm mode. 
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CHAPTER 11 


Logging and Monitoring 





Docker includes several built-in logging drivers for containers, such as json-file, syslog, journald, gelf, 
fluentd, and awslogs. Docker also provides the docker logs command to get the logs for a container. Docker 
1.13 includes an experimental feature for getting a Docker service log using the docker service logs 
command. 


The Problem 


Docker Swarm mode does not include a native monitoring service for Docker services and containers. 

Also the experimental feature to get service logs is a command-line feature and required to be run per service. 
A logging service with which all the services' logs and metrics could be collected and viewed in a dashboard 
is lacking. 


The Solution 


Sematext is an integrated data analytics platform that provides SPM performance monitoring for metrics 
and events collection, and Logsene for log collection, including correlation between performance metrics, 
logs, and events. Logsene is a hosted ELK (Elasticsearch, Logtash, Kibana) stack. Sematext Docker Agent 
is required to be installed on each Swarm node in the Swarm for continuously collecting logs, metrics, and 
events, as illustrated in Figure 11-1. 
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Swarm 
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SematextAgent 












Docker 
Containers SPM 


(Metrics & Events) 


Logsene 
(Logs) 


Figure 11-1. Sematext Docker agent on each Swarm node 


This chapter covers the following topics: 
e Setting the environment 
e Creating a SPM application 
e Creating a Logsene application 
e Deploying the Sematext Docker agent as a service 
e Creating a MySQL database deployment on Docker Swarm 
e Monitoring the Docker Swarm metrics 


e Getting Docker Swarm logs in Logsene 


Setting the Environment 


Start a three-node Swarm consisting of one manager and two worker nodes using Docker for AWS. (This is 
discussed in Chapter 3.) Obtain the public IP address of the manager node instance from the EC2 console 
and SSH login into the instance. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.227.123.67 
Welcome to Docker! 
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The procedure to use Sematext SPM and Logsene for logging and monitoring with a Docker Swarm is as 
follows. 


Create an account at https://apps.sematext.com/ui/registration. 


Log in to the user account at https: //apps.sematext.com/ui/login. 


Select the integrations (Logsene app and SPM Docker app) from https: //apps. 
sematext.com/ui/integrations?newUser, as listed in Steps 4 and 5. 


Create a SPM (a performance monitoring app). An app is like a namespace for 
data. A SPM token is generated that is to be used to install a Sematext agent on 


each Swarm node. 


Create a Logsene app. A Logsene token is generated that is also used to install a 
Sematext agent on each Swarm node. 


Install a Sematext agent on each Swarm node. Docker Swarm metrics, logs, and 
events start getting collected in the SPM dashboard and the Logsene dashboard. 


Creating a SPM Application 


Log in to a Sematext account at https://apps.sematext.com/ui/integrations?newUser to display the 
Integrations page. For a SPM Docker app, select Docker from Infrastructure and Application Performance 
Monitoring. In the Add SPM Docker App dialog, specify an application name (DockerSwarmSPM), as shown in 
Figure 11-2. Click on Create App. 


rh ey 


® uH 


Application name 


DockerSwarmSPM 
Discount code 
| Educatio i titution 


Invite team members 


Enter emails separated by 





Add SPM Docker App 


Non-profit? Small startup? Get in touch later for discount. 


| FE: 
Comma 


s 


Figure 11-2. Adding a SPM Docker app 
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An SPM App is created, as shown in Figure 11-3. Several client configurations are listed. 


To start monitoring follow the steps below on each host. 





> 
?5 
zz If you are using Docker Cloud to host your Docker containers, you can instead just press the button below, and paste your SPM token 9b5552fd-001d-44f0-9452-760464d4a3413 into the Docker Cloud 
template. 

LJ 





Client Configuration 


Requirement: Docker v1.6 or newer installed (or v1.12 in case of Docker Swarm) 
Linux / Mac OS X / Windows Coreos Kubernetes RancherOS Mesos Docker Swarm 


1. Start SPM Docker Agent container (this container with SPM Agent will be used to monitor all Docker containers located on the same host) 


sudo docker rum -d --name sematext-agent --restart=always -e SPM TOKEN»90555243-0010-44f0-9452-76046042a3413 -v /:/rootfs:ro -v /var/run/cocker.sock:/var/run/docker.sock sematext/cenatext - agen 


Figure 11-3. SPM app is created 


Click on the Client Configuration tab for Docker Swarm, as shown in Figure 11-4. The Docker Swarm 
tab displays the docker service create command to create a service for a Sematext Docker agent; copy the 
command. The command includes a SPM TOKEN, which is unique for each SPM app. 


Client Configuration 


Requirement: Docker v1.6 or newer installed (or v1.12 in case of Docker Swarm). 
Linux / Mac OS X / Windows CoreOS Kubernetes RancherOS Mesos Docker Swarm 


1. The following configuration will activate Sematext Docker Agent on every node in the Swarm Cluster cluster 


docker service create --mode global V 

--restart-condition any \ 

--name sematext-agent-docker X 

--mount typesbind,srcs/var/run/docker.sock,dsts/var/run/docker.sock X 

--mount typesbind,srcs/,dsts/rootfs,readonlystrue X 

-e SPM TOKEN-s9b5552fd-001d0-44f0-9452-760464d4a3413 sematext/sematext-agent-docker 


Q Waiting for data. Configure monitoring agent 





Figure 11-4. Docker Swarm configuration 
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The SPM app is added to the dashboard, as shown in Figure 11-5. Click on the App link to navigate to 
App Reports, which shows the monitoring data, metrics, and events collected by the SPM app and the charts 


generated from the data. 





& 
Al Apps © 
Eh Gasht 
Bil in Str 
T 
: i 
= Logs Type Application Token Plan State Your Role Data Received Actions 
xt Ae eockerSwarm5 e 
d int - 
Apps 


Figure 11-5. DockerSwarmSPM app on the dashboard 


As the message in Figure 11-6 indicates, the app has not received any data yet. All the metrics graphs are 
empty initially, but they will display the graphs when data starts getting received. 


ol num E E & * 


The DockerSwarmsSPM App has not received any data yet, wait a few more minutes or check troubleshoot € f x 
= 
lIÉ 
. Ensure you can connect to SPM Recerver. https; //scae-receiver .senatext.con/receiver P setungs may be needed, frewall may be king outbound traffic, or D id 
du missing required entries. See how to check network connectivity. 
e Make sure the disk where SPM agent is installed is nct ful 
= e Check if you issed any of the steps ir stal at inst 
e Please see known issues and troubleshooting wik 
x e It nothing else helped. please contact us directly via live chat or via email 
r3 
m 
& 
Ma Hos i nta im í a m 
There is no data for this chart. Check There is no data for this chart. Check There is no data for this chart. Check 
SPM troubleshooting guide SPM troubleshooting guide SPM troubleshooting guide 
Ma Container Count = mtainer Memory Fail Counte Ms Container Swap 
= ———— 


Figure 11-6. The DockerSwarmSPM app has not received any data 


Creating a Logsene Application 


To create a Logsene app, select Logs App from the integrations page at https: //apps.sematext.com/ui/ 
integrations?newUser, as shown in Figure 11-7. 
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< 
db Dashboards A You have no Apps for your logs yet. Create a new App and enjoy your logs! 


= infrastructure ` 
laa Monitoring - 
E au Monitoring Apps 


DockerSwarmsPM 





Docker 





os [ Create new Logsene app } 
Custom Metrics 

= Logs 

2G Correlations 


FE Alerts & Events 


d Integrations = 


Figure 11-7. Selecting the Logs app 


In the Add Logsene App dialog, specify an application name (DockerSwarmLogsene) and click on Create 
App, as shown in Figure 11-8. 


< 
Add Logsene App 


dò Dashboards 


= infrastructure Application name 


là Monitoring - | DockerSwarmLogsene 


E au Monitoring Apps 
Discount code 


DockerSwarmSPM 
Educational institution? Non-profit? Small startup? Get in touch later for d 

Docker + 

) invite team members 

os + 

; : Enter emails separated by comma 
Custom Metrics " 
E 
= Logs 


JG Correlations 


Create App 





E3 Alerts & Events 


& Inte grations - 


Figure 11-8. Adding the Logsene app 
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A new Logsene application called DockerSwarmLogsene is created, as shown in Figure 11-9. Copy the 
LOGSENE TOKEN that's generated, which we will use to create a Sematext Docker agent service in a Docker 





Docker 


We made it super simple to collect logs with Docker. 


kí Setup 


Add "=e LOGSENE TOKEN-81ac5395.-fe8f-47d9-93b2-dc00c649116a* to the Docker run 


into Logsene. 


The following will work with any Docker host. This will work with RancherOS as well 


Metot Marathor 


Cloud laas / Paas 


Mobile 


ios 


Android 


sudo docker pull sematext/sematext-agent-docker 
È sudo docker rum -d --name sematext-agent --restarteslneys | 
-e LOGSENE TOX£N«81ac5355- e64-4725-9352-dc00c6491168 \ 
-¥ /ver/run/eotker .£oCk: /var/tun/ecocker. sock \ 
^y /etc/loceltine: /etc/1ocaltise:ro ' 
Sematext/senatext - agent -docker 


At the same time, you can also collect Docker Metrics and Events and send them to SPM for Docker monitoring and alerting by adding"-e SPM_TOKEN= 


«YOUR SPM TOKEN»* to the Cocker run command. To get an SPM Token create SPM app for Docker 


Figure 11-9. The Logsene app is added and LOGSENE TOKEN is generated 


& 


A new Logsene application called DockerSwarmLogsene is added to the dashboard, as shown in 
Figure 11-10. 
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Figure 11-10. 
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Your Role Data Received 
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command of the Sematext Docker Agent to get all container and host logs 


Actions 
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Click on the DockerSwarmLogsene app link to display the log data collected by the app. Initially, the app 
does not receive any data, as indicated by a message in Figure 11-11, because we have not yet configured 


a Sematext Docker agent service on the Docker Swarm. The Logsene Ul is integrated with the Kibana 
dashboard. 


> e + > 
- The DockersswarmLogsene App has not received any data vet, wait a few more minutes or check troubleshooting tips: x 
= 
a Double-check your log shipping configuration, esp. any errors messages log shipper itself may be logging to its own log files or console icicle 
{i a Try running your log shipper m debug and/or verbose mode nday 
= Check if there are any network connectivity insues, see our wik page = 
= = Check that your servers" clocks are correct and check that your logs have accurate timestamps and not timestamps from the future -type 
+ Check that your logs have correctly formatted timestamps. Here are some examples of acceptable timestamps: 
x æ Elasticsearch API: 2001-06 -D8 TOR; C0:01.11237 (508601) or 1487854172123 (milliseconds since UTC epoch) zd 
+ Syslog: 2001-06-08TOB00c0131232 (S508601) of Oct 11 22714715 (RF C3164 timestamp - UTC) werelan 
E: For more details around timestamp formats sse wiki page 7 
& » Check integrations guide. once more or consider using a different log shipper TAGS 


See more details on how to troubleshoot Logsene appe on Logeene troubleshooting wiki 
If after all of the above you still don't see your logs please contact us via mail or via our ire chat. So we can help you faster, please let us know 
which log shipping method you are using. 


bal Log Counte 


No date found for the selected time period 


Figure 11-11. The app does not receive any data at first 


Connecting the SPM and Logsene Apps 


Next, connect the SPM and Logsene apps so that the metrics and events collected by the SPM are integrated 
with the Logsene app. Choose Integrations > Connected Apps, as shown in Figure 11-12. 


T o 
Dockerswarnms oM 





^ YOU can connect any two apps you have access to, regardless of their type. 
Dacke A single app can be connected to any number of other apps. 
Second app 
05 Why connect apps? 
Custom Matris = connecting a logs app to a monitoring app will pre-select that Logs app when you 
decide to correlate metrics fram the connected monitoring app, and thus save you 
= Loge - timi. 
= when you receive an alert notification for an app, the alert notification will 
E au Logs Apps automatically inzlude information (e.g. charts) from connected apps, and thus provide 
more information and context for you. 
Docker&warmLogsene 
: Connected apps 
Xi Correlations 
a ; à i - 
F3 Alerts & Events t 5 * ord : filka 
& integrations E : 
First App Second App Created On Created By Actions 
Overview There ig no data available 
Apps 


Connected Apps dy 


Hotheansn Heoks 


Figure 11-12. Choosing Integrations > Connected Apps 
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Select DockerSwarmSPM as the first app and DockerSwarmLogsene as the second app, as shown in 
Figure 11-13. Then click on Connect Apps. 


< First app €» 


DockerSwarmSPM Doci y 
ockerSwarmsPM NS ¥ è . e the 
ckers : You can connect any two apps you have access to, regardless of their type 


jocker À single app can be connected to any number of other apps 


o 
v 


Why connect apps? 
* connecting a logs app to a monitoring app will pre-select that logs app when you 
decide to correlate metrics from the connected monitoring app, and thus save you 


C storm ! 5 
= Logs ° time 
A * when you receive an alert notification for an app, the alert notification will 


G arc £s Apps automatically include information (e.g. charts) from connected apps, and thus provide 
more :nformation and context for you 
ockerSwarmLogsene 


Connected apps 
X Correlations 


FS Alerts & Events 5 v of O " 
& integrations B : ^ 
+ First App Second App Created On Created By Actions 
Overview There is no data available 
Connected Apps 
. 
Figure 11-13. DockerSwarmLogsene 
The connected apps are listed, as shown in Figure 11-14. 
< First app g$ 
DockerSwarmSPM a 
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Y apy 
Why connect apps? 
T | Me « connecting a logs app to a monitoring app will pre-select that Logs app when you 
decide to correlate metrics from the connected monitoring app. and thus save you 
= Logs v onnect apps time 
« when you receive an alert notification for an app, the alert notification will 
(B ait Logs Apps automatically include information (e.g. charts) from connected apps, and thus provide 
s more information and context for you 
mi. ^ 
| 
Connected apps 
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F2 Alerts & Events | 5 v ofi 
| 
ntegrations i | 
S — First App Second App Created On Created By Actions 
Overview DockerSwarmS PM DockerSwarmLogsene 2017-07-27 00:18:17 Jvofwal7Myahoo,.cor 
b 
Connected Apps 





Figure 11-14. The connected apps 


Deploying the Sematext Docker Agent as a Service 


The docker service create command copied earlier includes just the SPM TOKEN token. Add -e LOGSENE TOKEN 
obtained from the Logsene app. Run the docker service create command on the Swarm manager node. 


~ $ docker service create --mode global \ 

--restart-condition any \ 

--name sematext-agent-docker \ 

--mount type-bind,src-/var/run/docker.sock,dst-/var/run/docker.sock ^ 

--mount type-bind,src-/,dst-/rootfs,readonly-true \ 

-e SPM TOKEN-9b5552fd-001d-44f0-9452-760464d4a3413 \ 

-e LOGSENE TOKEN-81ac5395-fe8f-47d9-93b2-dc00c649116a \ 
sematext/sematext-agent-docker 

oubjk53mpdnjgak5dgfdxs4ft 


Vv VV VV VV 
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A service for the Sematext Docker agent is created; it's listed using docker service ls. 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE PORTS 
oubjk53mpdnj  sematext-agent-docker global 3/3 sematext/sematext-agent-docker:latest 


List the service tasks. As this is a global service, one task gets started on each node. 


~ $ docker service ps sematext-agent-docker 


ID NAME IMAGE 

NODE DESIRED STATE CURRENT STATE ERROR PORTS 

5jvl7gnvlOote | sematext-agent-docker.8dOqviepqu8xop402f9418j40 sematext/sematext-agent- 
docker: latest 

ip-172-31-8-4.ec2. internal Running Running 2 minutes ago 

y53f20d3kknh sematext-agent-docker.xks3sw6qgwbcuacyypemfbxy j sematext/sematext-agent- 
docker: latest 

ip-172-31-31-117.ec2.internal Running Running 2 minutes ago 


t5w2pxy4fc9l | sematext-agent-docker.rO2ftwtp3n4mocl7v211w4gi8 ^ sematext/sematext-agent- 
docker:latest 
ip-172-31-44-8.ec2.internal Running Running 2 minutes ago 


If additional nodes are added to the Swarm, the Sematext Docker agent starts a service task on the new 
nodes. As an example, update the CloudFormation stack to increase the number of manager nodes to three 
and worker nodes to five, as shown in Figure 11-15. 


(D CloudFormation v Stacks > Stack Detail > Update Stack 
Select Template Specify Details 
| Specify Details 
Options Specify parameter values. You can use or change the default parameter values, which are defined in the AWS CloudFormation template. Learn more 
Stack name D 
Parameters 
Swarm Size 
Number of Swarm 
managers? 
[> Number of Swarm worker 


nodes? 


Figure 11-15. Increasing the number of worker nodes 
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The Swarm nodes are increased to three manager nodes and five worker nodes when the Stack update 
is complete. 


~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
8d0qviepqu8xop40o2f9418j40 ip-172-31-8-4.ec2.internal Ready Active 
9rvieyqnndgecagbuf73r9gs5 ip-172-31-35-125.ec2.internal Ready Active Reachable 
j4mg3fyzjtsdcnmr7rkiytltj ^ ip-172-31-18-156.ec2.internal Ready Active 
mhbbunhl358chahidmrOy6i71 ip-172-31-7-78.ec2.internal Ready Active Reachable 


ro2ftwtp3n4mOcl7v21lw4gi8 ip-172-31-44-8.ec2.internal Ready Active 
vdamjjjrz7a3ri3prv9fjngvy ip-172-31-6-92.ec2.internal Ready Active 
xks3sw6qgwbcuacyypemfbxyj * ip-172-31-31-117.ec2.internal Ready Active Leader 
xxyy4ys40030bb4l5daoicsr2 ip-172-31-21-138.ec2.internal Ready Active 


Adding nodes to the Swarm starts a Sematext agent on the nodes that were added. 


~ $ docker service ps sematext-agent-docker 


ID NAME 

IMAGE NODE DESIRED STATE 
CURRENT STATE ERROR PORTS 

cgaturw05p59 sematext-agent-docker.xxyy4ys40030bb415daoicsr2 


sematext/sematext-agent-docker:latest ip-172-31-21-138.ec2.internal Running 
Running 2 minutes ago 

1j4f46q3ydv1 sematext-agent-docker. j4mg3fyzjtsdcnmr7rkiytltj 
sematext/sematext-agent-docker:latest ip-172-31-18-156.ec2.internal Running 
Running 2 minutes ago 

v54bjs3c8u5r sematext -agent-docker.vdamjjjrz7a3ri3prv9fjngvy 
sematext/sematext-agent-docker:latest ip-172-31-6-92.ec2.internal Running 
Running 2 minutes ago 

s7arohbeoake sematext-agent-docker.9rvieyqnndgecagbuf73r9gs5 
sematext/sematext-agent-docker:latest ip-172-31-35-125.ec2.internal Running 
Running 3 minutes ago 


ixpri65xwpds sematext-agent-docker.mhbbunhl358chahidmroy6i71 
sematext/sematext-agent-docker:latest ip-172-31-7-78.ec2.internal Running 
Running 4 minutes ago 

5jvl7gnvlote sematext-agent-docker.8dOqvi1epqu8xop402f9418j40 
sematext/sematext-agent-docker:latest ip-172-31-8-4.ec2.internal Running 
Running 15 minutes ago 

y53f20d3kknh sematext-agent-docker.xks3sw6qgwbcuacyypemfbxyj 


sematext/sematext-agent-docker: latest ip-172-31-31-117.ec2.internal Running 
Running 15 minutes ago 

t5w2pxy4fc9ol sematext-agent-docker.rO2ftwtp3n4mocl7v211w4gi8 
sematext/sematext-agent-docker:latest ip-172-31-44-8.ec2.internal Running 
Running 15 minutes ago 
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Creating a MySQL Database Service on a Docker Swarm 


In this section, we create a MySQL database service from which metrics, logs, and events can be collected 
with Sematext SCM and Logsene using the Sematext Docker Agent, which we installed. To start, run the 
following command to create a mysql service with 10 replicas. 


~ $ docker service create ^ 
--env MYSQL ROOT PASSWORD- 'mysql'N 
--replicas 10 \ 
--name mysql ^ 

mysql 

rmy45fpa31twkyb3dowzpc74a 


> 
> 
> 
> 


The service is created and listed in addition to the Sematext Docker agent service. 


~ ¢ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 

oubjk53mpdnj sematext-agent-docker global 8/8 sematext/sematext-agent- 
docker: latest 

rmy45fpa3itw mysql replicated 10/10 mysql:latest 


The service tasks for the mysql service are also listed. 


~ $ docker service ps mysql 


ID NAME IMAGE NODE DESIRED STATE 
CURRENT STATE ERROR PORTS 

x8j221ws4kx2 mysql.1 | mysql:latest ip-172-31-21-138.ec2.internal Running 
Running 13 seconds ago 

98rbd6nwspqz mysql.2 mysql:latest ip-172-31-44-8.ec2.internal Running 
Running 11 seconds ago 

vmqolylni8or mysql.3 | mysql:latest ip-172-31-8-4.ec2.internal Running 
Running 24 seconds ago 

Ovb6oda3yh3d mysql.4 mysql:latest ip-172-31-7-78.ec2.internal Running 
Running 23 seconds ago 

vdpplkyxyiuy mysql.5 | mysql:latest ip-172-31-6-92.ec2.internal Running 
Running 23 seconds ago 

9ser7fwz6998 mysql.6 mysql:latest ip-172-31-18-156.ec2.internal Running 
Running 17 seconds ago 

vfsfvanghnso mysql.7 | mysql:latest ip-172-31-18-156.ec2.internal Running 
Running 17 seconds ago 

v71qwpvjhhzn mysql.8 | mysql:latest ip-172-31-6-92.ec2.internal Running 
Running 23 seconds ago 

j7172i5ml43d mysql.9 | mysql:latest ip-172-31-31-117.ec2.internal Running 
Running 24 seconds ago 

5p5mg2wnbboo mysql.10 mysql:latest ip-172-31-35-125.ec2.internal Running 


Running 20 seconds ago 
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After the Sematext Docker agent service has been started on the Swarm and a MySQL database service 
has been started, both the SPM and Logsene apps start receiving data, as indicated by the Data Received 
column in the dashboard. See Figure 11-16. 


All Apps 
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D nerSwa SPM 9b55542f3 max XOCOX X , 445 Basic SPM Docker v 
= => - 
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Figure 11-16. DockerSwarmSPM overview 


Monitoring the Docker Swarm Metrics 


After the mysql service is started on the Swarm, the metrics for the service start getting loaded into the SPM - 
Performance Monitoring dashboard. This happens as soon as the Sematext Docker agent is installed and new 
metrics from a deployment become available. Graphs for different metrics—including Host CPU, Container 
CPU, Container Memory, Container Count, Container Memory Failed Counter, Container Swap, Container I/O 
Throughput, Container Network Traffic, and Container Network Errors—are displayed, as shown in Figure 11-17. 
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Figure 11-17. Docker Swarm SPM overview 
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The Docker container metrics—including Container Count, Container CPU, Container Disk, Container 
Memory, and Container Network—may be displayed by selecting Docker in the navigation. The Docker 
Container Count metrics are shown in Figure 11-18. 
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Figure 11-18. Docker metrics 


The Docker > Container Network selection displays the network traffic received and transmitted, the 
receive rate, and the transmit rate. The OS Disk Space Used may be displayed by choosing OS > Disk. The 
metrics collection granularity may be set to auto granularity (default), by month, by week, by day, by hour, by 
5 minutes, or by 1 minute. The Logs Overview may be displayed using the Logs button. 

Click the Refresh Charts button to refresh the charts if they are not set to auto-refresh, which is the 


default. 
Detailed logs are displayed using Logsene UI or Kibana 4, which we discuss in the next section. 


Getting Docker Swarm Logs in Logsene 


Select Logs > DockerSwarmLogsene in the margin navigation to display the logs collected by Logsene. 

The Log Counts, Log Events, and Filter fields are displayed, as shown in Figure 11-19. To search for logs 

generated by the mysql service, add “mysql” to the search field and click on the Search button. The logs 

generated by the mysql Docker service are displayed, including status messages such as “mysqld ready 
for connections". Click on the Refresh button to refresh the logs. 
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The Logsene collects all the Docker events, such as the Docker pull event for the mysql: latest image, 
as shown in Figure 11-20. 
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Figure 11-20. Logs for Docker event for mysql image pull 
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Logs for another Docker event, a volume mount, are shown in Figure 11-21. 
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Figure 11-21. Logs for Docker event volume mount 
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Summary 


This chapter discussed continuous logging and monitoring of a Docker Swarm with Sematext SPM 
performance monitoring and Logsene log management. First, you learned how to create a SPM app and a 
Logsene app. Then you installed a Sematext agent service on each of the Swarm nodes and monitored the 
metrics and events in a SPM dashboard. You also learned how to monitor the logs in the Logsene UI or a 
Kibana 4 dashboard. The next chapter discusses load balancing in a Docker Swarm. 
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CHAPTER 12 


Load Balancing 


A Docker Swarm mode service provides a distributed application that may be scaled across a cluster of 
nodes. Swarm mode provides internal load balancing among the different services in the Swarm based on 
the DNS name of a service. Swarm mode also provides ingress load balancing among a service's different 
tasks if the service is published on a host port. Additionally, service tasks may be scheduled on specific 
nodes using placement constraints. 


service Discovery 


A Swarm has a DNS server embedded in it. Service discovery is based on the DNS name. Swarm manager 
assigns each service in the Swarm a unique DNS name entry. Swarm manager uses internal load balancing 
to distribute requests for the different services in the Swarm based on the DNS name for a service. 


Custom Scheduling 


Service replicas are scheduled on the nodes in a Swarm using the spread scheduling strategy by default. 
A user may configure placement constraints for a service so that replicas are scheduled on specific nodes. 
Scheduling using constraints is discussed in Chapter 6. 


Ingress Load Balancing 


By default, each service that’s exposed on a published port for external access is added to the ingress 
overlay network. A user may specify any available port to expose a service by using the - -publish, or -p, 
option. The syntax for the - -publish (-p) option is --publish <PublishedPort>:<TargetPort> in which 
the <PublishedPort> variable is for the published port on the host and the <TargetPort> variable is for the 
container port. If the - -publish, -p option does not specify a <PublishedPort> port to publish the service 
on the Swarm, the manager automatically exposes the service on a published port chosen from the range 
30000-32767. 


The Problem 


Ingress load balancing is for distributing the load among the service tasks and is used even if a Swarm 
consists of a single node. Ingress load balancing for a multi-node Swarm is illustrated in Figure 12-1. A client 
may access any node in the Swarm, whether the node has a service task scheduled or not, and the client 
request is forwarded to one of the service tasks using ingress load balancing. 
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Figure 12-1. Ingress load balancing 


A single client accesses a single node and, as a result, the Swarm is under-utilized in terms of 
distributing external client load across the Swarm nodes. The client load is not balanced across the Swarm 
nodes. A single node does not provide any fault tolerance. If the node fails, the service becomes unavailable 
to an external client accessing the service at the node. 


The Solution 


An AWS Elastic Load Balancer (ELB) is used to distribute client load across multiple EC2 instances. When 
used for Docker Swarm mode an AWS Flastic Load Balancer distributes client load across the different EC2 
instances, which are hosting the Swarm nodes. The external load balancer accesses (listens to) the Swarm on 
each EC2 instance at the published ports for the services running in the Swarm using LB listeners. Each LB 
listener has an LB port mapped to an instance port (a published port for a service) on each EC2 instance. An 
ELB on a Swarm is illustrated in Figure 12-2. 
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Figure 12-2. External load balancer 


As a client is not accessing the service at a single host even if a single node goes down or becomes 
unavailable, the Swarm does not become unavailable as the external load balancer directs the client request 
to a different node in the Swarm. Even when all the nodes are available, the client traffic is distributed among 
the different nodes. As an example, a client could be being served from one node at a particular time and 
from a different node shortly thereafter. Thus, an external load balancer serves two functions: load balancing 
and fault tolerance. Additionally the cloud provider on which a Swarm is hosted may provide additional 
features such as a secure and elastic external load balancing. Elastic load balancing, as provided by AWS 
Elastic Load Balancer, scales the request handling capacity based on the client traffic. 

This chapter discusses load balancing with a user-created Swarm on CoreOS. It also discusses the 
automatically provisioned elastic load balancer on Docker for AWS managed services. 


Setting the Environment 


Start three CoreOS instances—one for the manager node and two for the worker nodes—as shown in 
Figure 12-3. Obtain the public IP address of the manager instance from the EC2 dashboard, as shown in 
Figure 12-3. 
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Figure 12-3. CoreOS instances on EC2 for a manager and two worker nodes 


SSH login into the manager node to initiate the Swarm mode. Initializing a Swarm mode on CoreOS 
and joining worker nodes to the Swarm is discussed in Chapter 2. Copy the docker swarm join command 
output to join the worker nodes to the Swarm. List the Swarm nodes with the docker node 1s command. 


core@ip-10-0-0-226 ^" $ docker node ls 

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 
9giqh5stg7hxy8u43tlifdiriOq ^ ip-10-0-0-203.ec2.internal Ready Active 
a0e1b2623qj03852mrc5cax97 + ip-10-0-0-198.ec2.internal Ready Active 
dsyo3b6553ueishozhfbiapad * ip-10-0-0-226.ec2.internal Ready Active Leader 


Creating a Hello World Service 


Next, create a hello world service with the docker service create command. Expose the service at port 
8080 using the - -publish option. The syntax to publish a service using - - publish or -p is as follows. 


docker service create \ 
--name <SERVICE-NAME> \ 
--publish <PUBLISHED-PORT>:<TARGET-PORT> \ 
«IMAGE» 


The <PUBLISHED-PORT> is the port exposed on the hosts and the «TARGET -PORT» is the port on which the 
Docker container exposes the service. Using the tutum/hello-world Docker image, «PUBLISHED-PORT» as 8080, 
«TARGET - PORT» as 80, and <SERVICE-NAME> as hello-world, run the following command to create the service. 


coreQip-10-0-0-226 ~“ $ docker service create ^ 


>  --name hello-world \ 
> | --publish 8080:80 \ 
»  --replicas 3 \ 


>  tutum/hello-world 
Ogk3wom7z91fpm5o9e6optmb5 
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The service is added to the ingress overlay network and the service is exposed at each node on the 
Swarm, whether a service task is running on the node or not. The hello-world service lists 3/3 replicas. 


core@ip-10-0-0-226 ~ $ docker service ls 
ID NAME REPLICAS IMAGE COMMAND 
Ogk3wom7z91f hello-world 3/3 tutum/hello-world 


List the service tasks using the docker service ps hello-world command and the three tasks are 
listed as scheduled, one on each node. 


core@ip-10-0-0-226 ~ $ docker service ps hello-world 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR 

di5oilh96jmr6fdshaevkkkt2  hello-world.1  tutum/hello-world ip-10-0-0-198.ec2.internal 
Running Running 24 seconds ago 

5g5d075yib2td8466mh7cO1cz hello-world.2 tutum/hello-world ip-10-0-0-226.ec2.internal 
Running Running 24 seconds ago 

Ssaarf4ngju3xr7uh7ninhooo hello-world.3 tutum/hello-world  ip-10-0-0-203.ec2.internal 
Running Running 23 seconds ago 


One Docker container is running on the manager node. 


core@ip-10-0-0-226 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

b73cbcd0c37e tutum/hello-world:latest ^ "/bin/sh -c 'php-fpm " 34 seconds ago 
Up 32 seconds 80/tcp hello-world.2.5g5d075yib2td8466mh7c01cz 


One Docker container is running on one of the worker nodes. 


core@ip-10-0-0-198 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

8bfiif2df213 | tutum/hello-world:latest ^ "/bin/sh -c 'php-fpm " 38 seconds ago 
Up 36 seconds 80/tcp hello-world.1.di50ilh96jmr6fd5haevkkkt2 


And the third Docker container is running on the other worker node. 


core@ip-10-0-0-203 ~ $ docker ps 


CONTAINER ID IMAGE COMMAND CREATED 

STATUS PORTS NAMES 

a461bfc8d4f9 — tutum/hello-world:latest ^ "/bin/sh -c 'php-fpm " 40 seconds ago 
Up 38 seconds 80/tcp hello-world.3.5saarf4ngju3xr7uh7ninhooo 
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Invoking the Hello World Service 


Without an external load balancer, an ingress connection may be made at each of the nodes at the published 
port. To invoke the service at the manager node, obtain the public DNS of the Swarm manager instance from 
the EC2 console, as shown in Figure 12-3. 

Invoke the service in a web browser at the «PublicDNS»:«PublishedPort» URL, as shown in Figure 12-4. 


' [3 Hello world! 


= CG | © ec2-52-91-212-27.compute-1.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is a461bfc8d4f9 





Figure 12-4. Invoking the service in a browser 
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Similarly, to invoke the service at a worker node, obtain the public DNS of the worker instance from the 
EC2 console and invoke the service in a web browser at the «PublicDNS»:«PublishedPort» URL, as shown 
in Figure 12-5. 


M Hello world! x 


c C | @ ec2-54-162-11-213.compute-1.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is b73cbcd0c37e 





Figure 12-5. Invoking the service at a worker node 
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Similarly, to invoke the service at the other worker node, obtain the public DNS of the worker instance 
from the EC2 console and invoke the service in a web browser at the <PublicDNS>:<PublishedPort> URL, as 
shown in Figure 12-6. 


[) Hello world! x 


S C | © ec2-34-201-118-146.compute-1.amazonaws.com:808( 


tutum 


Hello world! 


My hostname is a461bfc8d4f9 





Figure 12-6. Invoking the service at the other worker node 


While the external AWS Elastic Load Balancer distributes the load among the EC2 instances, the ingress 
load balancer distributes the load among the service tasks. In the preceding example, the same service task 
is invoked when the service is invoked at the Swarm manager instance and at a Swarm worker instance, as 
indicated by the same hostname (Figures 12-4 and 12-6). This demonstrates the ingress load balancing. 

A different service task could get invoked if the service is invoked at the same host. As an example, 
invoke the service at the Swarm manager instance again. A different service task is served, as indicated by a 
different hostname in Figure 12-7. This is in comparison to the hostname served earlier in Figure 12-4, again 
demonstrating the ingress load balancing. 


226 


CHAPTER 12... LOAD BALANCING 


[' Hello world! x 


| € Q | @ ec2-54-162-11-213.compute-Lamazonaws.com:8080 


tutum 


Hello world! 


My hostname is 8bf11f2df213 





Figure 12-7. Different hostname served when invoking the service at the manager node again 


Creating an External Elastic Load Balancer 


In this section, we create an external elastic load balancer on the AWS cloud. Click on Load Balancers in the 
EC2 dashboard. Then click on Create Load Balancer to create a new load balancer, as shown in Figure 12-8. 


Create Load Balancer Actions Y 
Em oeo 


Filter: Q Search x < < None found 


You do not have any of these resources in this region 


Select a load balancer mms m 


Figure 12-8. Creating a new load balancer 
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AWS Elastic Load Balancing offers two types of load balancers—classic load balancers and application 
load balancers. The classic load balancer routes traffic based on either application or network level 
information whereas the application load balancer routes traffic based on advanced application-level 
information. The classic load balancer should suffice for most simple load balancing of traffic to multiple 
EC2 instances and is the one we use for Docker Swarm instances. Select the Classic Load Balancer and then 
click on Continue, as shown in Figure 12-9. 


Welcome to Elastic Load Balancing 


Application Load Balancer * Classic Load Balancer 


eo Preferred for HTTP/HTTPS 





An Application Load Balancer makes routing decisions at the application layer A Classi Load Balancer makes routing Cecisions at either the transport layer 
(HTTP/HTTPS), supports path-based routing, and can route requests to one or (TCP/SSL) or the application layer (HTTP/HTTPS), and supports either EC2- 
more porns on each EC2 instance or container instance in your VPC Classic or a VPC 


Figure 12-9. Selecting the classic load balancer option 


In the Define Load Balancer dialog, specify a load balancer name (HelloWorldLoadBalancer) and select 
a VPC to create the load balancer in, as shown in Figure 12-10. The VPC must exist prior to creating the load 
balancer and must be where the EC2 instances to be load balanced are created. The load balancer protocol 
is HTTP and so is the instance protocol, by default. Keeping the default setting of HTTP protocol, specify the 
load balancer port and the instance port as 8080, because the Hello World service is exposed at port 8080. 


1. Define Load Balancer = 2. Assign Security Groups — 3 Configure Security Settings d. Configure Health Check — SAG ECZ instances — 6. Add Tags T. Review 


Step 1: Define Load Balancer 
Basic Configuration 


This wizard will walk you through setting up a new load balancer Begin by giving your new load balancer a unique name so that you can dentis it from other load balancers you might create, rou will also need tà 
configure ports and protocols for your load balancer, Traffic from your clients can be routed from any load balancer port to any port on your EC2 instances, Ey default, we've configured your load balancer with a 
standard web server on part &0. 


Load Balancer name: —Hello'WordLoadBalancer 


Create LB Inside: = |vpc-18&cea261 (10.0.0.0/24) | redshift-vpc 7 
Create an intemal load balancer: O (Qwharssha?) 
Listener Configuration: 
Load Balancer Protocol Load Balancer Port Instance Protocol Instance Port 
Nin | 080 HTTP , oso — e 


Add 


Select Subnets 


You will need to selecta Subnet Tor each Availability Zone where you wish kafie to be routed by your load balancer I you have instances in only one Availability Zone, please select al least two Subnets in 
dierent Availability Zones to provide higher availability tor your load balancer = 


Cancel Next: Assign Security Groups 


Figure 12-10. Selecting the load balancer protocol 
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In the Select Subnets tab, click on one or more subnets listed in the Available Subnets table. The subnets 
are added to the selected subnets, as shown in Figure 12-11. Click on Next. To provide high availability, 
select at least two subnets in different availability zones. 


1. Define Load Balancer 2. Assign Secumy Gaoups E nügurne S ty Settings +, Configure Health Check 5, Add EC? inslarces 6, Add Tag T, Beg 


Step 1: Define Load Balancer 


Load Balancer Protocol Load Balancer Port Instance Protecol Instance Port 
HTP : 8080 HTTP r 8080 ae 
Add 


Select Subnets 


You will need fo selecta Subnet Ter each Avallabilty Zone where you wish traffic to be routed Ey your load balancer. If you have instances in only one Availability Zone, please select al least two Subnets in 
different Availability) Aone To provide higher availabe Tor your bad balancer 


VPC vpc-15c6a261 (10.0.0 0/24) | redshift-vpc 
Available subnets 


Actions Availability Zone Subnet ID Subnet CIDR Name 


Selected subnets 


Actions Availability Zone Subnet ID Subnet CIDR Name 
o ugasi ia Submet-45318441 10,0 02s subnat-1 
e us-east-Tb subnet-ecddd cU 10.0.0 123/25 subnet-2 


Cancel Next: Assign Security Groups 


il 


Figure 12-11. Selecting subnets 


In the Assign Security Groups tab, select Create a New Security Group, as shown in Figure 12-12. In 
Type, select Custom TCP Rule. Choose the TCP protocol and the port range as 8080. Select Anywhere for the 
source and its value as 0.0.0.0/0. Click on Next. 


Define Load Balancer i. Assign Security Groups 3. Configure Secunty Settings 1. Configure Health Che: 5 Add EC? instances 6. Add Tags 7 Rewigs 


Step 2: Assign Security Groups 
You have selected ihe option of having your Elastic Load Balancer inside of a VPC, which allows you tà disk mec uriy groups lo VOUr load balancer Please select the Security woups lo assign bo this load balancer 
This can be c named At BT time 

Assign a security group: = Ceme m nau security anon 


Select an existing security group 


Security group name: quic k-create- 1 
Description: quick-create-1 created on Thursday, July 27, 2017 at 12:23:46 PM UTO- 
Type li Protocol |i Port Range (i Source |i 
Cuslom TCP Rule " TCP 8080 Anywhere + | 0.0.0.0/0 e 
Add Rule 


Cancel Previous Next: Configure Security Settings 


ke 


Figure 12-12. Assigning security groups 


Click on Next in Configure Security Settings, as we have not used the HTTPS or the SSL protocol. In the 
Configure Health Check tab, select HTTP for the ping protocol and 8080 for the ping port. Specify the ping path 
as /, as shown in Figure 12-13. Keep the defaults as is in the Advanced Details area and then click on Next. 
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Step 4: Configure Health Check 


Your load balancer will automatically perform health checks on your EC2 instances and only route traffic to instances that pass the health check. if an instance fails the health check, it is automatically removed from 


the load balancer. Customize the healih check to meet your specific needs. 
Ping Protocol HTTP M 
Ping Port $080 
Ping Path 


l 


Advanced Details 


Response Timeout <i 5 seconds 
interval <i 30 seconds 
Unhealthy threshold <i 2 M 
Healthy threshold (i 10 . 


Figure 12-13. Configuring a health check 


Cancel Previous 


Next; Add EC2 Instances 


Select the three Swarm instances listed, as shown in Figure 12-14. Also select Enable Cross-Zone Load 
Balancing, which distributes traffic evenly across all backend instances in all availability zones. Click on Next. 


Step 5: Add EC2 Instances 


The table below Ests all your running EC2 Instances. Check the boxes in the Select column to add those instances to this load balancer 


VPC vpc-18c63261 (10.0.0 0/24) | redsnift-vpc 


@ instance - Name M 
e (0106751 CoreOSWorker 
@ —i0a9e7ea5 CoreOSManager 
@  :(0dd912ca.. — CoreOSWorker 
Availability Zone Distribution 


3 Instances in us-east-1b 


% Enable Cross-Zone Load Balancing (i 


A Enable Connection Draining i) 300 seconds 


Figure 12-14. Adding EC2 instances 
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Security groups 

Container Linux by CoreOS -Stable--1409-7-0-AutogenBy. 
Container Linux by CoreOS -Stable--1409- 7-0-AutogenBy. 
Container Linux by CoreOS -Stable--1409-7-0-AutogenBy.. 


Zone 
us-east-1b 
us-east-1b 


us-east-1b 


*  SubnetiD 


subnet-ecdd4ic0 


subnet-ecdd41c0 


subnet-ecdd41c0 


Cancel 


Previous 


- 


Subnet CIDR 


10.0.0 128/25 
10.0.0 128/25 
10.0.0.128/25 


Next: Add Tags 


- 
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In the Add Tags tab, no tags need to be added. In the Review tab, click on Create, as shown in Figure 12-15. 
As indicated, the load balancer is an Internet-facing type. 


Step 7: Review 


+ Define Load Balancer 


Load Balancer name: 
Scheme: 
Port Configuration: 


* Configure Health Check 


Ping Target: 
Timeout: 

Interval: 

Unhealthy threshold: 
Healthy threshold: 


+ Add EC2 Instances 


Cross-Zone Load Balancing: 
Connection Draining: 
Instances: 


+ VPC Information 


Edit load balancer definition 


Hello WolkdLoadBalancer 
iMernet-facing 
£080 (HTTP) forwarding to 6080 (HTTP) 


Edit health check 


HTTP &DED/ 
5 seconds 
20 seconds 
3 

10 


EdR instances 


Enabled 
Enabled, 300 seconds 
i-Oddf12caasfatb1cf (CoreCSViorker), i-Da8e7eabSe*Sd'a6eD4 (CoreOSManager), i-DcTO&7b1fbcO3100f (CoreOSWorker) 


Edit subnets _ 


Cancel Previous. — 


Figure 12-15. Review your settings then create the load balancer 


A load balancer is created, as shown in Figure 12-16. 


Load Balancer Creation Status 


o Successfully created load balancer 
Load balancer HelloWorldLoadBalancer was successfully created 
Note: It may take a few minutes for your instances to become active in the new load balancer 





Figure 12-16. The load balancer has been created 
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Obtain the DNS name of the load balancer from the EC2 console, as shown in Figure 12-17. Initially, the 


status will be "0 of 3 instances in service" because the registration is still in progress. 


Create Load Balancer Actions v 


Filter: Q Search X 
Name * DNS name ~ State ~ VPCID 
Docker-ExternalLoa-1OWPS1IGIGSGT Docker-ExtemnalLoa-1OWPS vpc-f06f1a89 
@ = HelloWoddLoadBalancer HelloWordLoadBalancer-13... vpc-18¢6a261 


Load balancer: | HelloWorldLoadBalancer 
Description Instances Health Check Listeners Monitoring Tags 


Basic Configuration 


+ 


o9 0 


1t020f2 


Availability Zones ~ Type 


u$-east-1a, us-east-1b classic 


us-east-1a, us-east-1b classic 


Name: HelloWortdLoadBalancer Creation time: July 27, 2017 at 12:25:19 PM UTC-7 
N * DNS name: HelloWorldLoadBalancer-1388543093.us-east- Hosted zone: Z35SXDOTRO7X7K 
MEER Status: 0 of 3 instances in service 
Scheme: intemet-facing VPC: vpc-18c6a261 
Availability Zones: subnet-45310221 - us-east-12, 


subnet-ecdd41c0 - us-east-1b 


Figure 12-17. Obtaining the DNS name of the load balancer 


After a while, the status should become “3 of 3 instances in service" and all the instance should be 


InService, as shown in Figure 12-18. 


; Create Load Balancer Actions ~ 
EA octo 
Filter: Q Search X iK & 1to2012 > M 
Name = DNS name - Siate = VPCID =  Avallability Zones ~ Type 
Docker-ExtermalLoa-1OWPS 1IGIGSGT Docker-ExternalLoa-1OWPS vpc-füeftaB9 us-east-la. us-easi-Tb.... classic 
BU — HelloWeoldloadBalancer HelloWorldLoadBalancer-13... vpe-TBcb6a261 us-east-1a, us-easl-1b classic 
b 
Load balancer: | HelloWorldLoadBalancer BEE - 
Description instances Health Check Listeners Monitoring Tags 
Connection Draining: Enabled, 300 seconds (Edit) 
Edit Instances 
instance ID Name Availability Zone Status Ü Actions 
De 106 Tb ites 100f CoreOS Worker us-east-Tb Inservice (i. Remove from Load Balancer 
i-DaeTeabaoSd[a5e04 CoraeOSManager us-east-Tb InService (1) Remove from Load Balancer 
i-Qdd912caagfatbtcf Core OSWorker us-east- Tb InService C1) Remove from Load Balancer 


Figure 12-18. Status indicates three of three instances InService 
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The Hello World service may be invoked from the <DNSname> : «LoadBalancerPort» URL in a web 
browser, as shown in Figure 12-19. 


[ Hello world! x 


> Q | @ helloworldloadbalancer-1388543093.us-east-1.elb.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is b73cbcd0c37e 





Figure 12-19. Invoking the Hello World service 


The external elastic load balancer balances the load among the EC2 instances in the Swarm. Because 
the ingress load balancer balances the load among the different service tasks, a different service task could 
get invoked if the service is invoked at the ELB DNS name again, as shown in Figure 12-20. 
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[ Hello world! x 


€- Q | © helloworldloadbalancer-1388543093.us-east-1.elb.amazonaws.com:808C 


tutum 


Hello world! 


My hostname is 8bf11f2df213 





Figure 12-20. Different service task served 


Load Balancing in Docker for AWS 


While an external elastic load balancer had to be created when creating a Docker Swarm using the 
command line (by first initiating the Swarm mode and subsequently joining the worker nodes to the 
Swarm), the Docker for AWS managed service, which was introduced in Chapter 3, automatically 
creates an elastic load balancer. 

Create a Swarm (a Swarm created earlier may be updated) with three manager nodes and five worker 
nodes using Docker for AWS, as shown in Figure 12-21. An external elastic load balancer is created as one of 
the Swarm resources, as listed in the Resources tab in Figure 12-21. 
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(p CloudFormation ^ Stacks 


Filter: Active Showing 1 stack 
Stack Name Created Time Status Description 
&* Docker ZU17-D?-27 11:30c37 UTC-O700 UPDATE COMPLETE Docker CE for AWS 17.06. 0-ce (17.06. 0-ce-aws2) 2 
Overview Outputs Resources Events Template Parameters Tags — StackPolicy Change Sets BEE 
Logical ID Physical ID Type Status Status Reason 
i T 2563a(b88485eSadc a2 dbes5báf 

AZinfo MM [BLATEST]2862 5632008584565 adc a2dbe5b4!1 CistonicAZInfo CREATE COMPLETE 

AzZinfoFunction Docher-AzZInfoFunction-16Y 2 19 0 NTLOOG ANS: Lambda: Function CREATE_COMPLETE 

AtlachGaleway Cet he-Alllac-1KP 10x EGA4DPD4 AWS ECI VPC GalewayAllach CREATE_COMPLETE 

CioudstorEBSPolicy Decke-Clou-1GKBPE2NMOLIKUE AWS LAM: Policy CREATE COMPLETE 

DecherLogGroup Decker AWS Logs Legcroup CREATE COMPLETE 

UrynDBePolic ies Dicke DhynD-«CpBBACRBSHT Avv AME CIAM Policy CREATE COMPLETE 

DrynDBevwarkerPelicies Docke-DynD-1798 NW TOIL ANE AM POTY CREATE COMPLETE 

M ExtemalLoadBalancer Docker-ExtemalLoa-10WPS 11GIGSGT AWS ElasticLasdBalancing-:Lo CREATE COMPLETE 
External. oadBalancers 8g.18787e68 AWS EC2:-Securityisroup CREATE COMPLETE - 


Figure 12-21. CloudFormation stack for a Docker Swarm 


An Internet-facing Elastic Load Balancer is created, as shown in Figure 12-22. The public DNS for the 
load balancer may be used to access the Swarm, as discussed later. 


Create Load Balancer Actions v 
M reat Load Balancer oto 


Filter: Q Search x < < 1t010f1 
@ Name ~ DNSname ~ State ~ VPCID + Availability Zones ~ Type 
B)  Docker-ExternaiLoa-1OWPS1IGIGSGT Docker-ExternalLoa-1OWPS... vpc-f06f1a89 us-east-1a. us-east-1b, classic 


` 


Load balancer: | Docker-ExternalLoa-1OWP $11GIG5G T mmpmm: 


Description instances ^ Health Check | Listeners | Monitoring Tags 


Basic Configuration 


Name: Docker-ExternalLoa-1OWPS1IGIGSGT Creation time: July 27, 2017 at 11:31:49 AM UTC-7 
“DNS name: Docker-ExternalLoa-1OWPS1IGIGSGT- Hosted zone: Z35SXDOTRO7X7K 
449932458. uS-east-1.elb.amazonaws.com (A 


Status: 8 of 8 instances in service 
Record) 


VPC: vpc-f0611a89 
Scheme: Internet-facing 


Availability Zones: subnet-25422309 - us-east-1b, 
Subnet-72608816 - us-east-1a, 
subnet-d67b209e - us-east-1c 


Figure 12-22. Load balancer for the Swarm created with Docker for AWS 


Select the Instances tab. All the instances in the Swarm, manager or worker, are listed. All the instances 
should be InService, as shown in Figure 12-23. 
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Create Load Balancer Actions ¥ 


Filter: Q Search 


@ Name 


©  Docker-Externalloa-1OWPS1IGIGSGT 


Description Instances Health Check 


Connection Draining: Disabled (Ecit) 


Edit Instances 


Instance ID 
i-01b4#402d0a425099 
i-015d1c3f6e46149 11 
i-066¢a22a9340426 1d 
i-0ac60724422253013 
i-0ea799c5a529bd0f6 
i-Oed6788aa58926e 17 
i-05d0eec1b25989f533 
i-0918ccd5e57b537e7 


Figure 12-23. Instances status is InService 


x 


DNS name 


Docker-Manager 
Docker-Manager 
Docker-worker 
Docker-worker 
Docker-worker 
Docker-Manager 
Docker-worker 


Docker-worker 


Listeners 


State 


Docker-ExternalLoa-10WPS 


Monitoring Tags 


Availability Zone 
us-east-1b 
u$-east-1a 
us-east-1a 
us-east-1c 
u$-east-1c 
us-east-1c 
us-east-1b 


us-east-1a 


Status 


InService 
InSernce 
inService 
InService 
inService 
InService 
InService 


InSerece 


VPC ID 


vpc-f06f1a89 


o9 @ 


110 10f1 


Availability Zones ~ Type 


us-east-1a us-east-1b classic 


Actions 


Remove from Load Balancer 
Remove from Load Balancer 
Remove from Load Balancer 
Remove from Load Balancer 
Remove from Load Balancer 
Remove from Load Balancer 
Remove from Load Balancer 


Remove from Load Balancer 


Update the load balancer listeners in the Listeners tab to add/modify a listener with a load balancer 
port set to 8080 and an instance port set to 8080, which is the published port for the Hello World service we 
create, as shown in Figure 12-24. 


Create Load Balancer Actions “ 
q 


Filter: | C, Search 


@ Name 


B) Oocker-Externalloa-10VV/PS 1IGIGSGT 


X 


- DNS name 


- State 


Dacker-Extemalloa-1OWPS... 


Load balancer: | Docker-ExternalLoa-10WP 511G1G5G T 
Description Instances Health Check Listeners Monitoring 
The following listeners are cumently configured for this load balancer 


Load Balancer Protocol Load Balancer Port Instance Protocol 


TCP 8080 TCP 


b 


Edit 


Figure 12-24. The Listeners tab 


Tags 


Instance Fart 


8030 


Cipher 


MA 


SSL Certificate 


N/A 


o 9 @ 


1to012011 


- Availability Zones > Type 


us-east-la us-east-1b... classic 


Obtain the public IP address of one of the manager nodes from the EC2 console. 


SSH login to the manager node. 


[root@localhost ~]# ssh -i "docker.pem" docker@34.205.43.53 


Welcome to Docker! 
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List the Swarm nodes. 


~ $ docker node ls 

ID 
8d0qviepqu8xop402f9418 40 
8eckbotwpbuoslfr58lbibplh 
b6f18h4f3o44gkf5dhkzavoy3 
k9nl2zcmjzobbqusc5bkd829g 
pod70jwh5vpjwximcicpjfjkp * 
rO2ftwtp3n4mocl7v211w4gi8 
rd8dokksuts3aa07orhgkri3i 
xks3sw6qgwbcuacyypemfbxy j 


Create a Hello World service and expose the service at port 8080 (published port). 


~ $ docker service create \ 
--name hello-world \ 
--publish 8080:80 \ 


HOSTNAME 
ip-172-31-8-4.ec2.internal 
ip-172-31-32-133.ec2.internal 
ip-172-31-2-148.ec2. internal 
ip-172-31-21-41.ec2. internal 
ip-172-31-1-130.ec2. internal 
ip-172-31-44-8.ec2.internal 
ip-172-31-41-86.ec2.internal 
ip-172-31-31-117.ec2.internal 


STATUS 
Ready 
Ready 
Ready 
Ready 
Ready 
Ready 
Ready 
Ready 
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AVAILABILITY MANAGER STATUS 


Active 
Active 
Active 
Active 
Active 
Active 
Active 
Active 


LOAD BALANCING 


Leader 


Reachable 
Reachable 


VV VM 


--replicas 10 \ 
tutum/hello-world 


n4hmfognhjrasf5nhukr55krb 


Service tasks are scheduled across the Swarm. 


~ $ docker service ps hello-world 


ID NAME IMAGE 

DESIRED STATE CURRENT STATE ERROR 
yifetn3kpwwn hello-world.1 tutum/hello-world:latest 
Running Running 15 seconds ago 

5115zl9dickd hello-world.2 tutum/hello-world:latest 
Running Running 17 seconds ago 

k9glaavnogzg hello-world.3 tutum/hello-world:latest 
Running Running 17 seconds ago 

n83f89ijlokn hello-world.4 tutum/hello-world:latest 
Running Running 17 seconds ago 

nelf275h9tp1 hello-world.5 tutum/hello-world:latest 
Running Running 16 seconds ago 

w4c8zcvlq5v7 hello-world.6 tutum/hello-world:latest 
Running Running 17 seconds ago 

b5qvbbgkrpd5 hello-world.7 tutum/hello-world:latest 
Running Running 16 seconds ago 

qlm8dt9fuv92 hello-world.8 tutum/hello-world:latest 
Running Running 17 seconds ago 

t3tenhpahh7g hello-world.9 tutum/hello-world:latest 
Running Running 17 seconds ago 

up64ekxgeftk hello-world.10 tutum/hello-world:latest 
Running Running 17 seconds ago 


NODE 
PORTS 

ip-172-31-2-148.ec2. internal 
ip-172-31-44-8.ec2.internal 
ip-172-31-8-4.ec2. internal 
ip-172-31-41-86.ec2.internal 
ip-172-31-8-4.ec2.internal 
ip-172-31-32-133.ec2.internal 
ip-172-31-21-41.ec2. internal 
ip-172-31-31-117.ec2.internal 


ip-172-31-44-8.ec2.internal 


ip-172-31-1-130.ec2.internal 
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The hello-world service may be created without explicitly specifying a published port. 


" $ docker service create \ 
--name hello-world \ 
--publish 80 \ 
--replicas 3 \ 
tutum/hello-world 


VV VM 


The Swarm manager automatically assigns a published port in the range 30000-32767; the default being 
port 30000 if it’s available. The listener in the load balancer for the Docker for AWS Swarm may need to be 
modified to add a mapping for the LoadBalancerPort: ServiceInstancePort, such as 30000:30000. 

Obtain the public DNS for the elastic load balancer, which gets created automatically, as shown in 
Figure 12-25. 


Create Load Balancer Actions v 
La oto 


Filter: à Search X 1 to 1 of 1 

ig Name - DNSname ~ State ~ VPCID - Availability Zones - Type 
©  Docker-ExternalLoa-TOWPS 1IGIGSGT Docker-ExternalLoa-1OWPS vpc-f06f1389 us-east-1a. us-east-1b classic 

Load balancer: | Docker-ExternalLoa-1OWP S11GIG5G T mms m - 
Description Instances Health Check Listeners Monitoring Tags 


Basic Configuration 


Name: Docker-ExternalLoa-1OWPS1IGIG5GT Creation time: July 27, 2017 at 11:31:49 AM UTC-7 
* DNS name: Docker-ExternaiLoa-1OWPS1IGIGSGT- Hosted zone: Z35SXDOTRO7X7K 
M 449932458. us-east-1.elb.amazonaws.com (A MEME 
Status: 8 of 8 instances in service 
Record) 
VPC: vpc-f06f1a89 
Scheme: Internet-facing 
Availability Zones: subnet-25422309 - us-east-1b 


subnet-72608816 - us-east-1a 


subnet-d67b209e - us-east-1c 


Figure 12-25. Obtaining the public DNS of the ELB 


Access the service at <PublicDNS>:<PublishedPort> in a web browser, as shown in Figure 12-26. The 
request is forwarded to the ingress load balancer on one of the instances in the Swarm. The instance that the 
external request is forwarded to does not have to be hosting a service task. Finding a service task is what the 
ingress load balancer does. 
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[| Hello world! x 


a iS docker-externalloa-1owpsligig5gt-449932458.us-east-1.elb.amazonaws.com:8080 


tutum 


Hello world! 


My hostname is 5c339caaa9b8 





Figure 12-26. Accessing a Docker service at the elastic load balancer DNS 


Summary 


This chapter discussed load balancing in Swarm mode. An ingress load balancer is used to distribute 

the load among a service’s tasks. Each service in a Swarm is assigned a DNS name and an internal load 
balancer balances service requests among the services based on DNS name. We also created an external 
load balancer for AWS EC2 instances to distribute load among the EC2 instances. Docker for AWS creates 
an external load balancer automatically on AWS. In the next chapter we discuss developing a Docker Swarm 
based highly available website. 
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Developing a Highly Available 
Website 








High availability of a website refers to a website being available continuously without service interruption. 

A website is made highly available by provisioning fault tolerance into the Docker Swarm application. High 
availability is provided at various levels. The ingress load balancer balances incoming client requests across 
the multiple service tasks and provides fault tolerance at the tasks level. If one service task fails, client traffic 
is routed to another service task. Using an external load balancer for a Docker Swarm hosted across multiple 
availability zones is another method for providing high availability. An external load balancer provides fault 
tolerance at the node level. If one node fails, client traffic is routed to Swarm nodes on another node. 


The Problem 


Using an external load balancer such as an AWS Elastic Load Balancer provides fault tolerance across 
multiple availability zones in an AWS region. The elastic load balancer may be accessed at its DNS name by 
a Client host, as illustrated in Figure 13-1. The Swarm is not highly available, as failure of a single AWS region 
would cause a website to become unavailable. 


Availability Zone 1 





Load 
Balancer 


Availability Zone 2 


Availability Zone 3 


Figure 13-1. The elastic load balancer may be accessed at its DNS name by a client host 
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The Solution 


Amazon Route 53 provides high availability with various DNS failover options, including active-active and 
active-passive failover using alias resource record sets. Amazon Route 53 provides DNS failover across AWS 
regions that are geographically spread, as illustrated in Figure 13-2. We use the Amazon Route 53 active- 
passive failover configuration based on the primary-secondary architectural patter for load balancer DNSes. 


Region 
1 






Availability Zone 1 
Primary 
DNS 


Availability Zone 2 


AWS Route 53 
Hosted Zone 


Availability Zone 2 


Secondary 
DNS 
Availability Zone 3 


Figure 13-2. Amazon Route 53 provides DNS failover across AWS regions 


This chapter covers the following topics: 
e Setting the environment 
e Creating multiple Docker swarms 
e Deploying a Docker Swarm service 
e Creating a AWS Route 53 
e Creating a hosted zone 
e Configuring name servers 
e Creating record sets 
e Testing high availability 


e Deleting a hosted zone 
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Setting the Environment 


We use two Docker for AWS managed Swarms for providing two DNS for active-passive DNS failover 
configuration. A Route 53 provides the primary-secondary architectural pattern for the two DNSes. The only 
prerequisite is an AWS account, which may be created at https: //aws.amazon.com/resources/create- 
account/. Create a key pair (Swarm) that is to be used for SSH login to Swarm manager nodes, as shown in 
Figure 13-3. Set the permissions on the key pair to read-only by the owner only with the chmod 400 swarm. 
pem command. 





Resource Groups v '* [A . DeepakVohra » Oregon > Support v 
Create Key Pair Import Key Pair Delete 

AL e ooo 
Q 2) 1 to 1 of 1 
@ Key pair name ^ Fingerprint 
@ swam ae: ed:95:29 3c :0f:00 da fd: 5f d7:a1:c9: 30:29: be: 78: 6f:9b:e4 


Figure 13-3. Key pair 


A domain name must be registered to be used for creating an Amazon Route 53 hosted zone. 


Creating Multiple Docker Swarms 


Create two Docker Swarms using the Docker for AWS managed service at https: //docs.docker.com/ 
docker-for-aws/. The two Docker Swarms must be in two different AWS regions to use the high availability 
provided by geographically distributed AWS regions. Create one Docker Swarm Oregon region as an 
example, as shown in Figure 13-4. 


Services v Resource Groups ~  'k fA Deepak Vohra * — Oregon ~ — Support ~ 





@® CloudFormation ~ Stacks 





Create Stack T Actions v Design template | c o 

Filter. Active ~ Showing 1 stack 
| Stack Name Created Time Status Description 

* DockerSwarm-1 2017-02-12 14:01:27 UTC-0800 CREATE COMPLETE Docker for AWS 1.13.1 (ga-2) 


Figure 13-4. CloudFormation stack for Docker Swarm 


Each Docker Swarm has manager and worker nodes spread across the AWS availability zones in an AWS 
region. The public IP of a manager node may be obtained from the EC2 console, as shown in Figure 13-5. 
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Resource Groups + * D Deepak Vohra ~ Oregon + Support + 
Launch Instance Connect Actions v 
Codi o e ¢ 
Q | trib raaarchi by kovwor e 1to80f8 
Name ~ Instance ID a Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alam 
DockerSwarm-1-worker -D096516538a460342 tZ. micro us-west-2¢ @ running © 22checks... None 
DockerSwarm-1-worker -OS1913742454e9b90 — t2. micro us-west-2c @ running © 2/2 checks .. None 
@ —DockerSwarn-1-Manager i-D54193cb964bbbce5 — t2. micro us-west-2c Q running © 2/2 checks... None 
DockerSwarm-1-Manager i-D6c 7242048df94d06 t2.micro us-west-2b @ running © 22 checks... None. 
o b 
Instance: | i-054193cb964bbbce5 (DockerSwarm-1-Manager) Public DNS: ec2-54-149-86-148.us-west- m m m 


2.compute.amazonaws.com 





Description Status Checks Monitoring Tags 


Instance ID — i-054193cb864bbbce5 Public DNS (IPv4) — ec2-54-149-86-148. us-west- 
2 compute. amazonaws.com 
Instance state — running [>  IPv4 Public IP — 54.149.86.148 
Instance type — t2.micro IPv6IPs - 
Elastic IPs Private DNS — ip-172-31-44-164. us-west- - 


Figure 13-5. Obtaining the public IP of the Swarm manager node 


Using the public IP address for a manager node in the first Docker Swarm, SSH login to the manager 
node EC2 instance. 


[root@localhost ~]# ssh -i "swarm.pem" docker@54. 149.86.148 
Welcome to Docker! 
~$ 


Create the other Docker Swarm in the Ohio AWS region as an example, as shown in Figure 13-6. 
The regions may be different for different users. 


Services ~ Resource Groups ~v + 





(p CloudFormation ~ Stacks 


cate Stack " Actions * De sign template c o 
Fitter. Active > Showing 1 stack 
Stack Name Created Time Status Description 
+ DockerSwarm-2 2017-02-12 14:14:06 UTC-0800 CREATE COMPLETE Docker for AWS 1.13.1 (ga-2) 


» 


Figure 13-6. CloudFormation stack for the Docker Swarm in one region 
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The Swarm node EC2 instances for the second Docker Swarm are also spread across the AWS availability 
zones in the second AWS region, as shown in Figure 13-7. Obtain the public IP for a manager node. 


Launch Instance Connect Actions ¥ 
d 

q 
Name ~ Instance ID + Instance Type ~ Availability Zone ~ Instance State ~ 

B) — DockerSwarm-2-Manager i-009e6f31f168a1405a t2 micro us-east-2a Q running 
DockerSwarm-2-Manager i-01baf 37172925c 5a6 12. micro us-east-2b Q running 
DockerSwarm-2-worker i-0431b8d8e0f0343b5 12. micro us-east-2a Q running 
DockerSwarm-2-worker i-D5cf4eD1e909671f9 12 micro us-east-2c @ running 
DockerSwarm-2-worker i-D8609eb056b8522f7 t2 micro us-east-2a @ running 


Instance: | i-009e6f31f68a1405a (DockerSwarm-2-Manager) 


2.cO mpute.amazonaws.com 


Description Status Checks Monitoring Tags 


Instance ID — i-DO0Se6f31f68a1405a Public DNS (IPv4) 


IP«4 Public IP 
IPw6 IPs 


Instance stale — running 


Instance type — tZ. micro 


Figure 13-7. The Availability Zone column lists multiple zones 


SSH login to the instance. 


[root@1oca1.host —]# ssh -i "docker.pem" docker@52.14.23.163 
Welcome to Docker! 


"$ 
List the Swarm nodes in a Docker Swarm with the Docker node. 
~ $ docker node ls 


ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 


Public DNS: ec2-52-14-23-183.us-east- 


o 9 O 
1toBof8 

Status Checks ~ Alarm 
@ 2/2checks .. None 
© 22checks... None 
© 22checks.. None 
@ 2/2 checks None 

& 2/2 checks Mane ~ 
LI 

BEE - 


ec 2-52-14-23-163 us-easl- 
2 compute amaz ondas. com 


fncv7ducej3ind4u2sy9xtwi7 ip-172-31-34-223.us-east-2.compute.internal. Ready Active 


Reachable 


grdeu2x49yi2fmvuy9lmoogqg ip-172-31-43-174.us-east-2.compute.internal Ready Active 
keOd75qef9bg8t22eqv9spdpm ip-172-31-30-180.us-east-2.compute.internal. Ready Active 


Reachable 


m2mmifbrnjbdriub5r36zxyjc * ip-172-31-8-11.us-east-2.compute.internal Ready Active Leader 
qenbfrmsOxv7wom6wpw9yspw4 ip-172-31-27-178.us-east-2.compute.internal Ready Active 
tipzy29hgh3m6og5bzkgsego8 ip-172-31-12-37.us-east-2.compute.internal Ready Active 
v4xdl4jvthovrzsamujoxy3ju ip-172-31-7-219.us-east-2.compute.internal Ready Active 
vuq68yex58vzgx3audj3sm23a ip-172-31-28-182.us-east-2.compute.internal Ready Active 
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Deploying a Docker Swarm Service 


Next, we deploy a Hello World service that will be hosted on a website. Run the following command on a 
manager instance for the DockerSwarm-1 Swarm to create a tutum/hello-world service with two replicas 
exposed at port 8080 on the host nodes. 


docker service create \ 
--name hello-world \ 
--publish 8080:80 \ 
--replicas 2 \ 
tutum/hello-world 


A Docker service with two service tasks is created. 


~ $ docker service create \ 


> --name hello-world \ 

> --publish 8080:80 \ 

> -- replicas 2 \ 

> tutum/hello-world 
vn5F18h7t65s jwk54dwcoklhu 


~ $ docker service 1s 

ID NAME MODE REPLICAS IMAGE 

vn5tl8h7t65s hello-world replicated 2/2 tutum/hello-world:latest 

~ $ docker service ps hello-world 

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 

ac9ks5y9duni2 hello-world.l tutum/hello-worid:latest ip-172-31-19-220.us-west-2.compute. 
internal Running Running 13 seconds ago 

8s6r48wUui9 hello-world.2 tutum/hello-world:latest ip-172-31-24-250.us-west-2.compute. 


internal Running Running 13 seconds ago 


Scale the service to 10 replicas to provide load distribution. Subsequently, list the 
services to list 10/10 replicas as running." $ docker service scale hello-world-10 


hello-world scaled to 10 
" $ docker service ls 


ID NAME MODE REPLICAS IMAGE 
vn5U8h7t65s hello-world replicated 10/10 tutum/hello-world:latest 
~$ 


The 10 service task replicas are scheduled across the Swarm nodes, as shown in Figure 13-8. 
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~ $ docker service ps hello-world 


ID NAME 

DESIRED 
ac9ks5y9dum2 hello-world.1 
-2.compute.internal Running 
8s6r48wltui9 hello-world.2 
-2.compute.internal Running 
k4r20unv1lxxs hello-world.3 
-2.compute.internal Running 


imz825y6j5ya hello-world.4 
2.compute.internal Running 
m2lz9wpsbtea hello-world.5 


-2.compute.internal Running 
4kxqmmc3uxlw hello-world.6 
-2.compute.internal Running 
o3ychxa9ply8 hello-world.7 
-2.compute.internal Running 
uriyh6hh9026 hello-world.8 
-2.compute.internal Running 
nmrlnzv17yj5 hello-world.9 
2.compute.internal Running 
ghfduagrtgqg hello-world.10 
2.compute.internal Running 


~$] 


IMAGE 
STATE CURRENT STATE 
tutum/hello-world: Latest 


tutum/hello-world: Latest 


tutum/hello-world: Latest 
Running 49 seconds ago 
tutum/hello-world: Latest 
Running 46 seconds ago 
tutum/hello-world: Latest 
Running 45 seconds ago 
tutum/hello-world: Latest 
Running 46 seconds ago 
tutum/heLlo-world: Latest 
Running 46 seconds ago 
tutum/hello-world: Latest 
Running 46 seconds ago 
tutum/hello-world: Latest 
Running 46 seconds ago 
tutum/heLllo-world: Latest 
Running 45 seconds ago 


Figure 13-8. Service tasks scheduled across the Swarm nodes 


NODE 


ERROR PORTS 


1p-172-31-19-220.us-west 
Running about a minute ago 

ip-172-31-24-250.us-west 
Running about a minute ago 

1p-172-31-19-220.us-west 
1p-172-31-6-109.us-west- 
1p-172-31-44-164.us-west 
1p-172-31-42-245.us-west 
1p-172-31-36-249.us-west 
ip-172-31-20-251.us-west 
ip-172-31-3-209.us-west- 


1p-172-31-3-209.us-west- 


Obtain the load balancer DNS for the first Docker Swarm from the EC2 dashboard, as shown in Figure 13-9. 


Resource Groups v * 


LN Deepak Vohra v 


Oregon v Support v 





Create Load Balancer Actions Y 
4 


Filter: Q Search 


@ Name 


@ DockerSwarm-1-ELB 


Load balancer: | DockerSwarm-1-ELB 
Description instances 


Basic Configuration 


Name: DockerSwarm-1-ELB 


west-2 elb amazonaws.com (A Record) 


DNS name 


DockerSwarm-1-ELB-156265 


Health Check 


DockerSwarm-1-ELB-15626558885 us- 


State * VPCID 


Listeners Monitoring Tags 


Creation time: 


Hosted zone: 


VPC: vpc-287ce04f 


"DNS name: 
Scheme: internet-facing 
Availability subnet-18aa9e40 - us-west-2c 


Figure 13-9. Docker Swarm load balancer 


vpc -287ce04f 


Z1H1FLSHABSF5 


o 9 9 
1 to 1 of 1 


Availability Z: 


us-west-2a, us 


February 12, 2017 at 2:02:30 PM UTC-8 


Status: 8 of 8 instances in service 
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Access the service at «DNS» : «LoadBalancerPort» in a web browser, as shown in Figure 13-10; the load 
balancer port is set to 8080, the port at which the service is exposed. 





(3 Hello world! x 


€ C | © dockerswarm-1-elb-1562658885.us-west-2.elb.amazonaws.com Sy 


tutum 


Hello world! 





My hostname is d293e037e566 








Figure 13-10. Accessing the service in a browser 
Similarly for the second Docker Swarm, create a tutum/hello-world service with a published port set to 
8080. Scale the service to 10 replicas for load distribution across the Swarm. 
S docker service create \ 
> --name hello-world \ 
> --publish 8080:80 \ 
»> --replicas 2 \ 
> tutum/hello-world 
woqx2ltuibv53ctmuvssrsq8j 
~ $ docker service ls 
ID NAME MODE REPLICAS IMAGE 
woqx2ltuibv5 hello-world replicated 2/2 tutum/hello-world: latest 


~ $ docker service ps hello-world 
NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 
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ny9ermdgb7a4 hello-world.1 tutum/hello-world: latest ip-172-31-34-223.us-east-2.compute. 
internal Running Running 15 seconds ago 


Sw3thlgleinme hello-world.2 tutum/hello-world: latest ip-172-31-30-180.us-east-2.compute. 
internal Running Running 15 seconds ago 


~ ¢ docker service scale hello-world=10 


hello-world scaled to 10 


The service replicas are distributed across the Swarm nodes, as shown in Figure 13-11. 


~ $ docker service ps hello-world 


ID NAME 


IMAGE 


DESIRED STATE CURRENT STATE 


ny9ermdgb7a4 hello-world.1 

-2.compute.internal Running 
5w3thlglOmme hello-world.2 

-2.compute.internal Running 
lwarzpstn2wk hello-world.3 

.compute. internal Running 
ff58q94ij9l1m hello-world.4 

.compute. internal Running 
wtjskvuwhmu4 hello-world.5 

-2.compute.internal Running 
8vtnxa7yktxu hello-world.6 

-2.compute.internal Running 
Ombtqclkqwax hello-world.7 

2.compute.internal Running 
twfj8nn881u7 hello-world.8 

2.compute.internal Running 
98sykb376wn7 hello-world.9 

-2.compute.internal Running 
g6cyck25jd08 hello-world.10 
a"; eo Running 
aoe 


tutum/heLllo-world: Latest 


NODE 
ERROR PORTS 
1p-172-31-34-223.us-east 


Running about a minute ago 


tutum/hello-world:latest 


ip-172-31-30-180.us-east 


Running about a minute ago 


tutum/hello-world:latest 
Running 25 seconds 
tutum/hello-world:latest 
Running 25 seconds 
tutum/hello-world:latest 
Running 27 seconds 
tutum/hello-world:latest 
Running 26 seconds 
tutum/hello-world:latest 
Running 26 seconds 
tutum/hello-world:latest 
Running 26 seconds 
tutum/hello-world:latest 
Running 25 seconds 
tutum/hello-world:latest 
Running 26 seconds 


Figure 13-11. Service replicas distributed across the Swarm 


1p-172-31-8-11.us-east-2 


ago 
1p-172-31-8-11.us-east-2 
ago 
ip-172-31-30-180.us-east 
ago 
1p-172-31-43-174.us-east 
ago 
ip-172-31-12-37.us-east- 
ago 
ip-172-31-7-219.us-east- 
ago 
ip-172-31-27-178.us-east 
ago 


ip-172-31-28-182.us-east 
ago 
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Obtain the DNS of the elastic load balancer for the second Swarm, as shown in Figure 13-12. 





Resource Groups v Li A Deepak Vohra v Ohio+ Support v 
Create Load Balancer Actions Y 

. reate Load Balancer | oto 
Filter: Q Search x 1 to 1 of 1 
@ Name ~ DNS name ~ State ~ VPCID + Availability Z 
@  DockerSwam-2-ELB DockerSwarm-2-ELB-823944... vpc O1cf6fés uS-east-2a, us 

b 
Load balancer: | DockerSwarm-2-ELB mmm - 
Description Instances Health Check Listeners Monitoring Tags 


Basic Configuration 


Name: DockerSwarm-2-ELB Creation time: February 12, 2017 at 2:15:09 PM UTC-8 


"DNS name: DockerSwarm-2-ELB-823944047 us- Hosted zone: Z3AADJGXBKTTL2 


east-2 elb amazonaws.com (A Record) Status: ———H" 


Scheme: internet-facing VPC: pc-0O1cf6f68 
vpe- > 


Availability subnet-00259569 - us-east-2a 
Zones: subnet-0487437f - us-east-2b Y 





Figure 13-12. Obtaining the DNS name for the Swarm ELB 


Access the service at «DNS» : «LoadBalancerPort» in a web browser, as shown in Figure 13-13. 


Ú Bele werd! x 


e Q | (D dockerswarm-2-elb-823944047.us-east-2.elb.amazonaws.com:080 


tutum 


Hello world! 


My hostname is cc815a21b9e1 





Figure 13-13. Accessing the service in a browser 
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Creating an Amazon Route 53 


Amazon Route 53 is a highly available and scalable cloud Domain Name Service (DNS) web service that 
connects user requests to infrastructure running on the AWS, including Amazon EC2 instances, load 
balancers, and Amazon S3 buckets. We already created two Docker Swarms hosting the same Docker service 
using the Docker AWS managed service, which automatically creates an AWS ELB for each Docker Swarm. 

In this section, we create an Amazon Route 53 to route user requests to the nosqlsearch. com domain 
to the elastic load balancers for the two Docker Swarms. In Amazon Route 53, we create two resource 
record sets pointing to the two different ELBs configured for failover, with one of the ELBs being the primary 
resource record set and the other being the secondary resource record set. 

When the nosqlsearch.com domain is opened in a web browser, the Route 53 routes the request to the 
primary resource record set. If the primary record set fails, Route 53 routes the user request to the secondary 
record set, in effect providing high availability of the Hello World Docker service hosted on the nosqlsearch.com 
domain. To create an AWS Route 53, select Route 53 from the AWS services, as shown in Figure 13-14. 


E3 AWS Management Cons X 


c Q^ | @ Secure | https://us-east-2.console.aws.amazon.com 


! | Storage 
S3 
EFS 
Glacier 


Storage Gateway 


= Database 
RDS 
DynamoDB 
ElastiCache 
Redshift 


c? Networking & Content 
VPC 
CloudFront 
Direct Connect 


Route 53 


Č 


Figure 13-14. Selecting the Amazon Route 53 service 
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Creating a Hosted Zone 


A hosted zone is a configuration that determines how traffic to a domain on the Internet will be routed. To 
create a hosted zone, open https: //console.aws.amazon.com/route53/ in a web browser and click on 
Create Hosted Zone in the DNS management, as shown in Figure 13-15. 


| Dashboard DNS management 


Hosted zones 


Health checks A hosted zone tells Route 53 how 

to respond to DNS queries for a 
Traffic flow domain such as example.com 
Traffic policies Create hosted zone 
Policy records W 


Figure 13-15. Creating the hosted zone 


Alternatively, select Hosted Zones or open https: //console.aws.amazon.com/route53/hometthosted- 
zones in a browser and click on Create Hosted Zone, as shown in Figure 13-16. 


Create Hosted Zone Go to Record Sets Delete Hosted Zone 4 @ 


Dashboard 4 


| Hosted zones 







Health checks 
wmm 


o 


Traffic flow 
Traffic policies 


Policy records 


Domains Amazon Route 53 is an authoritative Domain Name System (DNS) service. DNS is the 
Registered domains system that translates human-readable domain names (example.com) into IP addresses 
Pending requests (192.0.2.0). With authoritative name servers in data centers all over the world, Route 53 is 


reliable, scalable, and fast. 


If you already have a domain name, such as example.com, Route 53 can tell the Domain Name System (DNS) where on the Internet to 
find web servers, mail servers, and other resources for your domain 
Learn More 


Create Hosted Zone 


Figure 13-16. Creating a hosted zone 
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Click on Create Hosted Zone again, as shown in Figure 13-17. 


Create Hosted Zone Go to Record Sets Delete Hosted Zone ou © 
x 
Dashboard 4 e 


Hosted zones Q <All Types . |4 € MoMostedZonestodisplay > >! 


Health checks 
Domain Name * Type * Record Set Count ~ Comment Hosted Zone ID 


Traffic policies 


Policy records 


Domains 
Registered domains 


Pending requests 
Figure 13-17. Creating a hosted zone 


In the Create Hosted Zone dialog, specify a domain name (nosqlsearch.com). The domain name must 
be registered with the user. Select Public Hosted Zone for the type, as shown in Figure 13-18. 


Create Hosted Zone Go to Record Sets Delete Hosted Zone E o 


Q ' field x All Types E Create Hosted Zone 


A hosted zone is a container that holds information about how you 
want to route traffic for a domain, such as example.com, and its 
subdomains 


I€ € NoHosted Zones to display > >»! 
Domain Name~  Type- Record Set Count- Comment 


Domain Name:  noscsearch com 


Comment: 


Type: public Hosted Zone Y 


A public hosted zone determines now traffic is 
routed on the Internet. 


Figure 13-18. Configuring the hosted zone 
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A new public hosted zone is created, as shown in Figure 13-19. The name servers for the hosted zone 


(by default, there are four) are assigned. 


Back to Hosted Zones Import Zone File 
4 
Q X | Any Type v Aliases Only 
Weighted Only 
Displaying 1 to 2 out of 2 Record Sets 
Name Type Value 


ns-1293.awsdns-33.org. 
ns-1929.awsdns-49.co.uk 


@  nosgisearch.com. NS 
ns-175.awsdns-21.com 
ns-538.awsdns-03.net 
nosqisearch.com SOA ns-175.awsdns-21.com. awsdns-hostmaster.am. 


" 


Figure 13-19. The new public hosted zone 


Configuring Name Servers 


d: Test Record Set 
Edit Record Set 
Name: nosqlsearch.com 
Type: NS - Name server 
Alias: 
TTL (Seconds): 1728 im | 5m th 
Value: 4. 1293 awsdns-33 org 
ns-1929.awsdns-49.co.uk 


ns-175 awsdns-21 com 

The domain name of a name server 
Enter multiple name servers on 
separate lines 

Example 
ns1 amazon.com 
ns2 amazon.org 
ns53 amazon.net 


n$4 amazon.co.uk 


Next, we need to configure the name servers for the domain with the domain registrar. The procedure to 
configure name servers is different for different domain registrars, but an option to add a zone record for a 


domain should be provided. 


Specify the record type as Nameserver, as shown in Figure 13-20. Specify the host as 0. Each zone record 
should point to a single name server, which may be obtained from the public hosted zone we created earlier. 
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Add Zone Record 


NOSQLSEARCH.COM 


Record type: * View current 
NS (Nameserver) 

Host: * @ 

@ 


Points to: * @ 


ns-1293 awsdns-33.org 
TTL: * G) Seconds: * 


Custom 


Cancel 





Figure 13-20. Adding a name server record 


Add four name servers (collectively called a delegation set), as shown in Figure 13-21, for the domain for 


which a hosted zone is to be created. 
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NOSQLSEARCH.COM ~ 


Status: Active | Created: 2012-07-28 | Expires: 2017-07-28 | Folder: None | Profile: None 


Q Renew v Q Upgrade $ Buy&Sel v 2 Account Change € Delete 





Settings DNS Zone File Contacts 


Auto-Renew (7) Standard: Of 
Extended: Of 


Manage 


Lock (7) On 


Manage 


Nameservers (7) NS-1293 AWSDNS-33.0RG 
NS-1929 AWSDNS-49.CO.UK 
NS-175.ANSDNS-21.COM 
NS-538 AWSDNS-03.NET 
Updated 2017-02-12 
Manage 


Figure 13-21. Name servers configured on a domain 


Creating Resource Record Sets 


After creating and configuring a hosted zone, create one or more resource record sets. A resource record set 
is a Domain Name System (DNS) configuration for routing traffic to a domain. Click on Create Record Set to 
create a resource record set, as shown in Figure 13-22. 
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Back to Hosted Zones -a SS 


Q, Record Set Name 
Weighted Only 
I< 
Name Type 
E nosgisearch.com. NS 
nosglsearch.com. SOA 


X | Any Type v Aliases Only 


Č Displaying 1 to 2 out of 2 Record Sets ^ >! 
Value 


ns-1293.awsdns-33.org. 
ns-1929 awsdns-49 co.uk. 
ns-175.awsdns-21.com. 
ns-538 awsdns-03 net. 


ns-175 awsdns-21.com. awsdns-hostmaster am: 


Figure 13-22. Creating a record set 
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Delete Record Set Test Record Set 
Edit Record Set 
Name: nosgisearch.com 
Type: NS -Name server 
Alias: O Yes © No 
TTL (Seconds): 172800 | 1m || Sm || 1h || 1d | 
Value: | ns-1293.awsdns-33.0rg al 
ns-1929 awsdns-49 co.uk 7 
ns-175.awsdns-21.com. | 


The domain name of a name server 
Enter multiple name servers on 
separate lines 

Example: 
nsi amazon com 
n52 amazon.ong 
ns3 amazon. net 
nsá amazon.co.uk 


Save Record Set 


In the Create Record Set tab, the type should be set to A -IPv4 address, as shown in Figure 13-23. The 
name of each record set ends with the domain name. Select Yes for Alias. 





Back to Hosted Zones Create Record Set Import Zone File 
Q, Record Set Name X Any Type v Aliases Only 
Weighted Only 
|< << Displaying 1102 out of 2 Record Sets ^ >! 
Hame Type Value 
ns-1293.awsdns-33.org. 
ns-1929 awsdns-49.co. uk 
nosgisearch.com. NS 
ns-175.awsdns-21.com, 
ns-538 awsdns-03 net. 
nosqisearch.com. SOA ns-175.awsdns-21 com. awsdns-hostmaster.am: 


Figure 13-23. Configuring a record set 


Delete Record Set Test Record Set 


Create Record Set 


Name: nosgisearc h.com 
Type: A -= IPv4 address 7 
Alias: @) Yes © No 


Alias Target: Enter target name 


You can also type the domain name for the resource. Examples: 

=- CloudFront distribution domain name: d111111abcdefB cloudfront net 

- Elastic Beanstalk environment CHAME: example elasticbeanstalk com 

- ELB load balancer DNS name: example-1.u5-&6851- 1 &lb amazonaws com 
- 93 website endpoint example.z3-website-us-easi-1.amazonaws.com 

- Resource record set in this hosted tone: www example com 

Learn Mor& 


Routing Policy: Simple Y 


Route &3 responds to queries based only on the values in this record. Learn 
More 


Evaluate Target Health: | yes œ No 
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Next, select the alias target as the AWS Elastic Load Balancer DNS for one of the Docker Swarms, as 


shown in Figure 13-24. 





Back to Hosted Zones Create Record Set Import Zone File 
Q Record Set Name X Any Type v Aliases Only 
Weighted Only 
|€ € Displaying 1 to 2 out of 2 Record Sets ^ >! 
Name Type Value 
ns-1293.awsdns-33.org. 
ns-1929 awsdns-49 co.uk 


nosgisearch.com. NS 
ns-175.awsdns-21.com. 


ns-538 awsdns-03.net. 


nosqisearch. Com. SOA ns-175.awsdns-21 com. awsdns-hostmaster am; 


Figure 13-24. Selecting an alias target 


Next, select the routing policy, as shown in Figure 13-25. 





Back to Hosted Zones Create Record Sel Import Zone File 
Q, Record Set Name X Any Type v Aliases Only 
Weighted Only 
|é € Displaying 1 to 2 out of 2 Record Sets > >! 
Name Type Value 
ns-1293.awsdns-33.org. 
ns-1929 awsdns-49.co.uk 
nosgisearch.com. NS 
ns-175.awsdns-21.com. 
ns-538 awsdns-03.net. " 
nosgisearch com. SOA — ns-175 awsdns-21 com. awsdns-hostmaster. am; 
IB k 


Figure 13-25. Selecting a routing policy 
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Delete Record Set Test Record Set 


Create Record Set 


Name: nosqisearch.com 
Type: A = [Pv address Y 
Alias: @ Yes ON 


Alias Target: = E 


You can also type — c3 website endpoints — 


. CloudFront distr 
T l 
. Elastic Beanstaj NO Targets Available 


- ELB load balanc — ELS Application load balancers — 
- S3 website endi No Targets Available 
~ Resource recor — ELE Classic load balancers — 
Learn Mor& 
HelloWorndLB-138358012.u$s-&ast-1.elb.ames 
, Dock my ?-ELB-823944047_us-east-2. 
Routing Policy erswa E D J 
Dockerswarm-1-ELE-1562658885.us-west-; 
Route 63 respond — /^waEmn? nieirihiminne — 
More 


Evaluate Target Health: © Yes 


un Na 


Delete Record Set Test Record Set 


Create Record Set 


Name: nosgisearc h.com 
Type: A = [Pw address Y 
Alias: @ Yes O No 


Alias Target: dualstack DockerSwarm-2-ELB-82394 
Alias Hosted Zone ID: Z3AADJGXGKTTLŽ 


You can also type the domain name for the resource. Examples: 

- CloudFront distribution domain name: d111111abodef8. cloudtiront net 
- Elastic Beanstalk environment CHAME: example elasticbeanstalk. com 
- ELB load balancer ONS name: example-1.us-east- 
1.elb.amazonaws.com 

- 53 website endpoint example 53-webaite-us-east-1 amazonaws com 
- Resource record set in this hosted Zone: www example. com 

Leam More 












Routing Policy: Simple 
i imple 

Route 53 responds fo q ; in this record. 
LARES SUE Weighted 


Latency 
Failover 
^| Geolocation 
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Select Failover for the routing policy. This configures DNS failover, as shown in Figure 13-26. Select 
Failover Record Type as Primary. 


Back to Hosted Zones Create Record Set 

















Import Zone File Delete Record Set Test Record Set 
: =} Create R rd Set ee 
q Record Set Name x Any Type v Aliases Only IE 
Name: nosgisearch.com 
Weighted Only 
Type: A -= IPv4 address 
|« € Displaying 1 to 2 out of 2Record Sets > >! | 
Alias: © Yes O No 
Name Type Value 
Alias Target: dualstack DockerSwarm-2-ELB-82394 
n$-1293.awsdns-33.0rg. Alias Hosted Zone ID: Z3AADJGX6KTTL2 
nosgisearch.com. NS UN THE aieeaa aii You can also type the domain name for the resource. Examples: 
ns-175.awsdns-21.com. - CloudFront distribution domain name: d1111118bcdef8 cloudfront net 
ns-538 awsdns-03 net. ù - Elastic Beanstalk environment CNAME: example elasticheanstalk com 
- ELB load balancer ONS name: example-1.us-east- 
nosgisearch.com. SOA ns-175.awsdns-21.com. awsdns-hostmaster am; | 1-4 mazonaws.com 
. 53 website endpoint example &3-webhsite-us-aast-1 amazenaws oom 
= Resource record set in this hosted zone: www example com 
Learn More 


Routing Policy: Failover "| 


Route 53 responds fo queries using primary record sets if any are 
healthy, OF using secondary record sets otherwise. Leam Moré 


| Dave Record Type: @) Primary _ O Secondary | 5x 
203 oom + | 


Figure 13-26. Selecting failover record type 
For Evaluate Target Health, select Yes, as shown in Figure 13-27. 


Back to Hosted Zones 


Create Record Set Import Zone File Delete Record Set Test Record Set 
i Alias Target: istack. k .2- hs 
Q Record Set Name X Any Type v Aliases Only dualstack.DockerSwarm-2-ELB-82394 
Alias Hosted Zone ID: Z3AADJGX6KTTL2 
Weighted Only You can also type the domain name for the resource. Examples: 
- CloudFront distribution domain name: d111111abodef8 cloudfront net 
- Elastic Beanstalk environment CNAME: example elasticbeanstalk. com 
I< < Displaying 1 to 2 out of 2 Record Sets > >| - ELB load balancer DNS name: example-1.us-east- 
1.eib amazonaws com 
Name Type Value - $3 website endpoint: example.s3-website-us-east-1 amazonaws.com 
- Resource record set in this hosted zone: www.example.com 
ns-1293.awsdns-33.org. Learn More 
re NS ns-1929 awsdns-49 co.uk 
ns-175.awsdns-21.com. Routing Policy: Failover v 
ns-538.awsdns-03.net. 


Route 53 responds to queries using primary record sets if any are 
healthy, or using secondary record sets otherwise. Learn More 
nosqisearch.com. SOA . ns-175.awsdns-21.com. awsdns-hostmaster am; 


Failover Record Type: «€ Primary © Secondary 


Set ID: Primary 


Evaluate Target Health: pa O No 


Associate with Health Check: — Yes 


« E , 


Figure 13-27. Selecting the Evaluate Target Health option 


ie; No 
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For Associate with Health Check, select No. Click on Create, as shown in Figure 13-28. 





Back to Hosted Zones Create Record Set Import Zone File Delete Record Set Test Record Set 
: i Alias Target: Istack. = x, : = 
CQ. Record Set Name x Any Type v Aliases Only get dualstack.DockerSwarm-2-ELB-82394 
Alias Hosted Zone ID: Z3AADJGXSKTTL2 
Weighted Only You can also type the domain name for the resource. Examples: 


- CloudFront distribution domain name: d111111abodef8. cloudfront net 
= Elastic Beanstalk environment CNAME: example. elasticbeanstalk com 
- ELS bad balancer ONS name: example-1. us-east. 

1.elb. amazonaws com 


[E © Displaying 11020utof 2 Record Sets > P| 


Name Type Value - $3 website endpoint: example 23-website-us-east-1 amazonaws.com 
- Resource record set in this hosted Zone: www example. com 
ns-1293.awsdns-33.org. Leam More 
m NS ns-1929 awsdns-49 co.uk 
x | ns-175.awsdns-21.com. Routing Policy: | Failover , 
na-538.awsdns-03 net. T 


Route 53 responds to queries using primary record sets if any are 
healthy, or using secondary record sets othenwise, Leam More 
nosgisearch.com. SOA — ns-175.awsdns-21. com. awsdns-hostmaster.am: 


Failover Record Type: © Primary ©) Secondary 


Set ID: Primary 


Evaluate Target Health: @ yes O No 


Associate with Health Check: | Yes © No 
Figure 13-28. Creating a record set 


A primary record set is created, as shown in Figure 13-29; “primary” implies that website traffic will be 
first routed to the record set. 


Back to Hosted Zones Import Zone File Delete Record Set Test Record Set 
Q Record Set Name X| Any Type + Aliases Only 


@To get started, click Create Record Set button or click an 
Weighted Only existing record set. 


|* € Displaying 1to030utof 3 Record Sets > >! 


Name Type Value 
N nosgisearch.com. — A ALIAS dualstack dockerswarm-2-elb-823944047 
ns-1293 awsdns-33 org. 
ns-1828.awsdns-49.co.uk. 
nosqisearch.com. NS T 
n$-175.awsdns-21.com. 
ns-538.awsdns-03.net 


nosqlsearch.com. SOA ns-175.awsdns-21 com. awsdns-hostmaster arm; 


Figure 13-29. Primary record set 
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To create a secondary record set, click on Create Record Set again, as shown in Figure 13-30. 


Back to Hosted Zones enn Import Zone File Delete Record Set Test Record Set 


Q Record Set Name X Any Type * Aliases Only 
@To get started, click Create Record Set button or click an 


Weighted Only existing record set. 


I€ € Displaying 1 to 3 out of 3 Record Sets > >! 
Name Type Value 
nosqisearch.com. A ALIAS dualstack.dockerswarm-2-elb-823944047 
ns-1293.awsdns-33.org 
— RM ns-1929.awsdns-49.co.uk : 
ns-175.awsdns-21.com 


ns-538.awsdns-03.net. 


nosqisearch.com. SOA . ns-175.awsdns-21.com. awsdns-hostmaster am; 


Figure 13-30. Creating another record set 


Select the type as A -IPv4 address and choose Yes for Alias. Select Alias Target as the second ELB DNS, 
as shown in Figure 13-31. 





Back to Hosted Zones Create Record Set Import Zone File Delete Record Set Test Record Set 
Q Record Set Name X. Any Type v Aliases Only ——Üa 
Name: nosqisearch.com 
Weighted Only 
Type: A - IPv4 address v 
|« © Displaying 1 to 3 out of 3 Record Sets > >! 
Alias: € Yes © No 
Name Type Value 
Alias Target: 
nosqisearch.com A ALIAS dualstack dockerswarm-2-elb-823944047 You cen also type| — 53 website endpoints — z 
ns-1293.awsdns-33.0rg. d cro AL No Targets Available 
aaa. (RB ns-1929.awsdns-49.co.uk. , ^ ELB load baland — ELS Application load balancers — 
ns-175.awsdns-21.com. - S3 website end; No Targets Available 
ns-538.awsdns-03 net. VILE — ELB Classic load balancers — a 
HelloWorldLB- 138359012.us-east-1.elb.amz 
nosqisearch.com SOA ns-175.awsdns-21.com. awsdns-hostmaster.am: Routing Policy DockerSwarm-2-ELB-823944047 us-east-2.: 
C——— 
Route 53 respond lg. lnuetie mnt efietrihiitinns — = 
More 


Evaluate Target Health: |^ Yes © No 


Figure 13-31. Selecting an alias target 
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Select the Failover routing policy and the secondary Failover Record Type, as shown in Figure 13-32. 


Back to Hosted Zones Create Record Set Import Zone File 
Q Record Set Name X. Any Type v Aliases Only 
Weighted Only 


I€ © Displaying 1 to 3 out of 3Record Sets » >! 


Name Type Value 


nosqisearch.com. A ALIAS dualstack.dockerswarm-2-elb-823944047 


ns-1293.awsdns-33 org. 
ns-1929.awsdns-49.co.uk. 

nosqisearch.com. NS D 
ns-175.awsdns-21.com. 


ns-538.awsdns-03.net. 


nosqisearch.com. SOA . ns-175.awsdns-21.com. awsdns-hostmaster.am: 


< > 


Figure 13-32. Selecting failover record type as secondary 


Delete Record Set Test Record Set 


Create Record Set 


Name: nosqisearch.com 


Type: A - IPv4 address v 
Alias: * Yes © No 


Alias Target: dualstack DockerSwarm-1-ELB-15626 
Alias Hosted Zone ID: Z1H1FLS5HABSFS5 


You can also type the domain name for the resource. Examples: 

- CloudFront distribution domain name: d111111abcdef8. cloudfront net 
- Elastic Beanstalk environment CNAME: example.elasticbeanstalk.com 
- ELS load balancer DNS name: example-1.us-east- 
1.elb.amazonaws.com 

- S3 website endpoint: example $3-website-us-east-1.amazonaws.com 
- Resource record set in this hosted zone: www.example.com 

Learn More 


Routing Policy: Failover Y 


Route 53 responds to queries using primary record sets if any are 
healthy, or using secondary record sets otherwise. Learn More 


Failover Record Type: © Primary @ Secondary A 


Create 


Choose Yes for the Evaluate Target Health and No for the Associate with Health Check. Click on Create, 


as shown in Figure 13-33. 


Back to Hosted Zones Create Record Set Import Zone File 
Q Record Set Name X | Any Type v Aliases Only 
Weighted Only 


I€ € Displaying 1 to 3 out of 3Record Sets » Ðl 


Name Type Value 


nosqisearch.com. A ALIAS dualstack.dockerswarm-2-elb-823944047 
ns-1293.awsdns-33.0rg. 
ns-1929 awsdns-49.co uk. 

nosqisearch.com. NS " 
ns-175.awsdns-21.com. 
ns-538.awsdns-03.net. 


nosqisearch.com SOA ns-175.awsdns-21.com. awsdns-hostmaster am; 


: -Á > 


Figure 13-33. Creating a secondary record set 
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Delete Record Set Test Record Set 


Alias Target: dualstack DockerSwarm-1-ELB-15626 
Alias Hosted Zone ID: Z1H1FL5HABSF5 


You can aiso type the domain name for the resource. Examples: 

- CloudFront distribution domain name: d111111abcdef8 cloudfront net 
- Elastic Beanstalk environment CNAME: example. elasticbeanstalk.com 
- ELB load balancer ONS name. example-1.us-east- 
1.elb.amazonaws com 

- S3 website endpoint: example s3-website-us-east-1 amazonaws com 
- Resource record set in this hosted zone: www example.com 

Learn More 


Routing Policy: Failover M 


Route 53 responds to queries using primary record sets if any are 
healthy, or using secondary record sets otherwise. Learn More 


Failover Record Type: ©) Primary ™ Secondary 
Set ID: Secondary 
Evaluate Target Health: © Yes | No 


Associate with Health Check: | Yes © No 


^ 


4 
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The secondary record set is created; “secondary” implies that traffic is routed to the record set if the 
primary record set fails, as shown in Figure 13-34. Click on Back to Hosted Zones. 


Back to Hosted Zones Create Record Set Import Zone File Delete Record Set Test Record Set 


Q set Nan X Any Type v Aliases Only 


QI To get started, click Create Record Set button or click an 
Weighted Only existing record set 


I€ < Displaying 1104 out of 4 Record Sets > >»! 


Name Type Value 
nosqisearch.com A ALIAS dualstack.dockerswarm-2-elb-823944047 
nosqlsearch.com À ALIAS dualstack.dockerswarm-1-elb- 156265888 


ns-1293.awsdns-33.org 


E ns-1929.awsdns-49.Co.uk 
nosqisearch.com NS 
ns-175.awsdns-21.com 


ns-538 awsdns-03.net 


nosqisearch.com SOA ns-175.awsdns-21.com. awsdns-hostmaster.am: 


Figure 13-34. Secondary record set is created 


The domain (nosqlsearch. com) is configured with four record sets, as shown in Figure 13-35. 


Goto Record Sets Delete Hosted Zone 


Q Search all fekis X | All Types ' 


Q 
© 


Hosted Zone Details 


Domain Name: nosgisearch.com 


[€  * Displaying 1 to 1 out of 1 Hosted Zones ^ >| 
Type: Public Hosted Zone 


Domain Name ~ Types Record Set Count Comment Hosted Zone ID Hosted Zone ID: ZZWEJOWBFEX&US 
Record Set Count: 4 
© nosgisearch.com. Public 4 Z2WEJDWBFEXSBL 
Ds Comment: # 


Name Servers *: ns-175.awsdns-21.com 
n5-538 awsdns-03. net 
ns-1293 awsdns-33.0ng 
ns-1979 awsdns-49.c0 uk 


* Before the Domain Name System wif shart fo route 
guerres for Puis domain fo Roule 53 names servers vou 
must update the name server records either with fe 
currant OAS senvice or with tha registmr for me domain 
as applicabile. For more information, click Ehe ? icon 
above 


Figure 13-35. Hosted zone created 


Testing High Availability 


Next, we test the high availability we configured. Open the domain, including the service published port 
(nosqlsearch.com:8080), in a web browser, as shown in Figure 13-36. The Docker service output should be 
displayed. 
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[3 Hello world! x 


1 = CQ | © nosglsearchcom: S080 


tutum 


Hello world! 


My hostname is 1ed90aee6c33 





Figure 13-36. Invoking a service in a browser 


To test high availability, delete the CloudFormation stack for the Docker Swarm associated with the 
primary record set, as shown in Figure 13-37. 


(D CloudFormation v Stacks 


| Create stack | = mec dir design template c 9 


Filter Active v Create Change Set For Current Stack Showing 1 stack 
Update Stack 
Stack Name , Status Description 
¥ 
4 DockerSwarm CREATE_COMPLETE Docker for AWS 1 13 1 (ga-2) 


View/Eda template in Designer 





Figure 13-37. Deleting a stack 


Click on Yes, Delete in the Delete Stack dialog. The stack should start to be deleted, as indicated by the 
DELETE IN PROGRESS status shown in Figure 13-38. 


(D CloudFormation ~ Stacks 





VILE MERE | Actions + Design template C V9 
Filter. Active * Showing 1 stack 
Stack Name Created Time Status Description 
¥ DockerSwanm-2 2017-02-12 14:14:06 UTC-0800 DELETE IM PROGRESS Docker for AWS 1.13.1 (ga-2) 


Figure 13-38. The delete is in progress 
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The DNS fails over to the secondary resource record set and the domain continues to serve the Docker 
service, as shown in Figure 13-39. 


Hello word! 


| c X |(D nosqisearch.com:808 


tutum 


Hello world! 


My hostname is 1ed90aee6c33 





Figure 13-39. Domain continues to serve 


The hostname in the browser could become different if the request is forwarded to a different service 
task replica, as shown in Figure 13-40. But the hostname could also become different regardless of whether 
failover has been initiated, because the ingress load balancer distributes traffic among the different service 
replicas. 
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[] Hello world! x 


c C |(D nosglsearch.com: 080 


tutum 


Hello world! 


My hostname is 316082fe5308 








Figure 13-40. Different hostname 


Deleting a Hosted Zone 


Before a hosted zone can be deleted, all the resource record sets associated with the hosted zone must be 
deleted. Select the resource record sets to delete and click on Delete Record Set, as shown in Figure 13-41. 


Back to Hosted Zones Create Record Sel Import Zone File Delete Record Set lest Record Set 


Ù 


2 record sets selected 


q Record Set Name X Any Type v Aliases Only 
Selected resource records: 
Weighted Only 
Name Type 
IE E Displaying 1 to 4 out of 4 Record Sets ^ >I Isearch.com, A 
nosgisearch.com, A 
Name ^ Type“ Value 
@ nosqlisearch.com. A ALIAS dualstack.dockerswarm-2-elb-82394404 
@ nosqisearchcom. A ALIAS dualstack.dockerswarm-1-elb-15626588. 


ns-1293 awsdns-33.org 
ns-1929 awsdns-49.co.uk. 
ns-1/5.awsdns-21.com 
ns-538.awsdns-03.net 


nosqisearch.com. NS 


nosqisearch.com SOA ns-175.awsdns-21.com. awsdns-hostmaster.arr 


Figure 13-41. Deleting the record sets 
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Click on Confirm in the Confirm dialog, as shown in Figure 13-42. 


Confirm Cancel x 


Are you sure you want to delete the following 2 record 
sets? 


e nosqisearch.com. 
* nosqisearch.com. 


| Confirm | Cancel 


Figure 13-42. Confirmation dialog 


Click on Back to Hosted Zones, as shown in Figure 13-43. 





Create Record Set Import Zone File Delete Record Set 


C Record Set Name X Any Type * Aliases Only 


Test Record Set 


Q To get started, click Create Record Set button or click an 


Weighted Only existing record set. 
lë Ë Displaying 1 to 2 out of 2 Record Sets > >! 
Name ^ Type” Value 
ns-1293 awsdns-33 org. 


ns-1829 awsdns-49.co.uk. 
nosqisearch.com NS 


ns-175.awsdns-21.com 
ns-538.awsdns-03.net i 


nosqisearch.com. SOA ns-175.awsdns-21.com. awsdns-hostmaster.arr 


Figure 13-43. Going back to the hosted zones 


Select the hosted zone to delete and click on Delete Hosted Zone, as shown in Figure 13-44. 
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Go to Record Sets Delete Hosted Zone 


k : Hosted Zone Details 
Q Search all fields X. Al Types " 


Domain Name: nosqisearch.com. 
Displaying 1 to 1 out of 1 Hosted Zones 
K < » » Type: Public Hosted Zone 
Domain Name ~  Type* Record Set Count Comment  ! Hosted Zone ID: Z2WEJDWBFEX8US3 
Record Set Count: 2 
@ nosqisearch.com. Public 2 
Comment: # 


Name Servers *: ns-175.awsdns-21.com 
ns-538.awsdns-03.net 
ns-1293.awsdns-33.org 
ns-1929 awsdns-49 co.uk 


* Before the Domain Name System will start to route 
queries for this domain to Route 53 name servers, you 
must update the name server records either with the 
current DNS service or with the registrar for the domain, 
as applicable For more information, click the ? icon 
above 


Figure 13-44. Deleting a hosted zone 


Click on Confirm in the Confirm dialog, as shown in Figure 13-45. 


Confirm Cancel X 


The name servers for this hosted zone are still the name 
servers for the nosqisearch.com domain. 

If you want to use this domain name in the future, we 
recommend that you either keep the hosted zone or 
transfer DNS service to another provider to prevent future 
DNS queries from possibly being misrouted. 


Learn more 


Are you sure you want to delete the nosqisearch.com 
hosted zone? 


cm [Gest 


Figure 13-45. Confirmation dialog for deleting a hosted zone 


The hosted zone is deleted. 
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Summary 


This chapter developed a highly available website using an Amazon Route 53 hosted zone. First, we created 
two Docker Swarms using the Docker for AWS managed service and deployed the same Docker service on 
each. Each Docker Swarm service may be accessed using the AWS Elastic Load Balancer for the Docker 
Swarm created automatically by the Docker for AWS. The Route 53 hosted zone is to create a hosted zone for 
a domain to route traffic to DNSes configured in the primary/secondary failover pattern. Subsequently, we 
tested that if the Docker Swarm for the primary record set is shut down, the website is still available, as the 
hosted zone routes the traffic to the secondary ELB DNS. In the next chapter we discuss using the Docker 
Swarm mode in Docker Cloud. 
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CHAPTER 14 


Using Swarm Mode in Docker Cloud 


Docker for AWS is a managed service for Docker Swarm based on a custom Linux distribution, and 

hosted on AWS with all the benefits inherent with being integrated with the AWS Cloud platform, such as 
centralized logging with CloudWatch, custom debugging, auto-scaling groups, elastic load balancing, and a 
DynamoDB database. 


The Problem 


While AWS is a managed cloud platform, it is not a managed service for Docker containers, images, and 
services per se. Docker’s builds and tests still need to be integrated. 


The Solution 


Docker Cloud is a managed service to test code and build Docker images and to create and manage 
Docker image repositories in the Docker Cloud registry. Docker Cloud also manages Docker containers, 
services, stacks, nodes, and node clusters. A stack is a collection of services and a service is a collection of 
containers. Docker Cloud is an integrated cloud service that manages builds and images, infrastructure, 
and nodes and apps. 

Docker Cloud also introduced a Swarm mode to manage Docker Swarms. In Swarm mode, Docker 
Cloud is integrated with Docker for AWS. As a result, Docker Cloud Swarm mode is an integration of two 
managed services—Docker for AWS and Docker Cloud. 

Docker Cloud provides some Docker images to interact between a Docker Swarm and a Docker host 
client, as discussed in Table 14-1. 


Table 14-1. Docker Images for Docker Swarm 

Docker Image Description 

dockercloud/client Used on the client side to start an interactive shell to connect to a remote 
docker Swarm cluster using Docker ID credentials. 


dockercloud/client-proxy Used on the client side to forward local docker API calls to a remote 
swarm cluster by injecting Docker ID authorization information on 
each request. 


dockercloud/server-proxy Authenticates and authorizes incoming Docker API calls and forwards 
them to the local Docker engine. 

dockercloud/registration Registers a Swarm cluster to Docker Cloud and launches a server proxy. 
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In this chapter, we discuss the Docker Cloud Swarm mode to provision a Docker Swarm with 
infrastructure hosted on AWS. This chapter covers the following topics: 


e Setting the environment 

e Creating an IAM role 

e Creating a Docker Swarm in Docker Cloud 

e Connecting to the Docker Swarm from a Docker host 

e Connecting to the Docker Swarm from a Swarm manager 


e Bringing a Swarm into Docker Cloud 


Setting the Environment 


As Docker Cloud is a managed service, all that is required is an account, which may be created at 
https://cloud.docker.com/. An AWS account is also required and may be created at https: //aws.amazon. 
com/resources/create-account/. Also create a key pair in the region in which the EC2 instances for the 
Docker Swarm will run, as shown in Figure 14-1. 





Resource Groups ~ * [X Deepak Vohra * — Ohio Support ~ 
, Import Key Pair o e © 
Q o 1 to 2of2 
Key pair name a Fingerprint - 
correos Sf lkce: 08: 2b te aod? TF. faced Ge De 38:08 er 74:18; 72:31 
by doc ker je:a5:ea: 08: 30-40 ba: 7 X 04: 76:15:30: 35: 2c :d6:d4- dec 41:0c 44 


Figure 14-1. Creating a key pair on AWS EC2 


Creating an IAM Role 


The Docker Cloud Swarm mode requires an AWS role with a new policy, an embedded policy for Docker 
for AWS. To create the IAM role, navigate to https: //console.aws.amazon.com/iam/home?#roles in a web 
browser. Click on Create New Role, as shown in Figure 14-2. 





Services ~ Resource Groups ~  '* fA  DeepakVohra~ — Global ~ 
O 
Dashboard Showing 0 results 
Role Name > Creation Time > 


Credential report 


Figure 14-2. Creating a new role 
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Specify a role name (dockercloud-swarm-role), as shown in Figure 14-3, and click on Next Step. 


Services + Resource Groups ~ t a! Deepak Vohra = — Global - Support + 


dyt 





Create Role set Role Name 
Step 1: Set Role Name Enter à role name. You cannot edit the role name afer the role is created 


F Role Name | dackercloud-swarm-role | 


Figure 14-3. Specifying a role name 


The Select Role Type page is displayed, as shown in Figure 14-4. As we are linking two services— Docker 
Cloud and Docker for AWS—we do not need to select an AWS service role. 





Services v Resource Groups ~v * A Deepak Vohra ~ Global» Support v 


Create Role Select Role Type 
Step 1: Set Role Name * AWS Service Roles 


Step 2 : Select Role Type 


>» Amazon EC2 
, x Select 
Allows EC? instances to call AWS services on your behal 
>» AWS Directory Service 
Select 
Allows AWS Directory Service to manage access for existing directory users and groups to AWS services 
? AWS Lambda 
Select 
Allows Lambda Function to call AWS services on your behalf 
? Amazon Redshift 
i Select 
Allows Amazon Redshift Clusters to call AWS services on your behalf 
? Amazon API Gateway 
: Select 
Allows API Gateway to cat AWS resources on your behalf - 


Role for Cross-Account Access 


Role for Identity Provider Access 


Cancel Previous Next Step 


Figure 14-4. Select the role type 


Select Role for Cross-Account Access, as shown in Figure 14-5, and select the sub-choice called Provide 
Access Between Your AWS Account and a 3rd Party AWS Account using the Select button. 
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Services + Resource Groups + t FL Deepak ur Riiie EESTI i + 





Create Role Select Hole Type 
Step 1: Set Role Name AIS Service Roles 


Step 2 : Select Role Type [*]Role for Cross-Account Access 


^ Provide access between AWS accounts you own 


Select 
Allows LAM Sers Mom one ar your other AVS accounts T6 access INS account 
^ Provide access between your AWS account and a 3rd party AWS account 
ect 
Allows LAM users from a 3rd party AWS account be access this account and enforces use of Extemal ID cS 
LT 


Role for Identity Providar Access 


Figure 14-5. Role for cross-account access 


Next, specify the account ID of the third party AWS account whose IAM users will access the AWS 
account. A third-party AWS account has been set up for the Docker Cloud service and has an account ID of 
689684103426, which may be used by anyone (AWS user) linking Docker Cloud service to their AWS account. 
Specify the account ID as 689684103426, as shown in Figure 14-6. The external ID is a user's Docker ID for 
the Docker Cloud service account created at https: //cloud.docker.com/. While the account ID will be the 
same (689684103426) for everyone, the external ID will be different for different users. Keep the Require MFA 
checkbox unchecked. Click on Next Step. 


Services + Resource Groups + t al Deepak Vohra ~ Gaba ~ Suppt ~ 





Enter thé ID of the 2nd party AWS account whose LAM users wil be able to access this account Enter the éxtennal ID provided by the and party. For details, see 
About the External ID 


,reate Role 


Step 1: SH Role Mame 


Account ID: SH9684 103476 
Step 2° Select Role Type 
Step 3 : Establish Trust External ID: dvonra 
Require MFA: 


Cancel Previous Next Step jy 


Figure 14-6. Specifying account and external IDs 
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As we are embedding a custom policy, do not select from any of the listed policies in Attach Policy. Click 
on Next Step, as shown in Figure 14-7. 


Services + Resource Groups ~v k ja Decpak Vohra ~ Global~ Support ~ 





Create Role Attach Policy 


Step 1: Set Role Name Select one or more policies to attach. Each role can have up to 10 policies attached 
Step 2 : Select Role Type 

Son + h Tnic 

Step 3 . Establish Trust Filter: Policy Type v Showing 269 results 
Step 4 : Altach Policy 


Policy Name $ Attached Entities > Creation Time > Edited Time > 

_| WW» AdministratorAccess 0 2015-02-05 10:39 PDT 2015-02-06 10:39 PDT 
 ) WW  AmazonAPIGatewayAdministr 0 2015-07-09 10:34 POT 2015-07-09 10:34 PDT 
(] WR  AmazonAPIGatewaytnvokeFul 0 2015-07-09 10.36 PDT 2015-07-09 10.36 PDT 
Ld [T] AmazonAPiGatewayPusn ToCl 0 2015-11-11 15:41 PDI 2015-11-11 15:41 POT 
E Gi —AmazonAppStreamFult'Access 0 2015-02-06 10:40 POT 2015-02-06 10:40 PDT 
J] WW»  AmazonAppSireamReadOnyA 0 2015-02-06 10.40 POT 2016-12-07 13:00 PDT 

| Gi AmazonAppStreamServiceAcc 0 2016-11-18 20:17 PDT 2016-11-18 20°17 PDT 
—) WW» AmazonAthenaFullAccess 0 2016-11-30 08:46 PDT 2016-11-30 08:46 POT 
_) WW»  A^mazonCioudOkectoryFullAcc 0 2017-02-24 16.41 POT 2017-02-24 16.41 POT 
xti "T San aen tendi ee t^ n ane? fo ^0 46-49 nint 143 665 ^0 «£40 ANT ix: 


cen | envious REIR 


Figure 14-7. Do not select a policy 


On the Review page, click on Create Role, as shown in Figure 14-8. 


Services ~v Resource Groups ~v $ A Deopak Vohra ~ Global ~ Support ~ 





reque To Review 
Step 1. Set Role Name Review the following role information To edit the role, click an edit link, or click Create Role to 
n "e finish 
Step 2 . Select Role Type 
Role Name  dockercioud-swarm-role Egit Role Name 
Step 3. Establish Trust 
» Role ARN amm aws.iam.. 672593526585 role/dockercloud 
Step 4 . Attach Policy CREATE 
Step $ ; Review Trusted Entities The account 689684103426 
Policies Change Policies 
Give this link to https //signin.aws. amazon.com’ swichrole? Copy Lini 


users who can account=$725935266554roleName=dockerc! 
switch roles in the  oud-swarm-role 
console 


Cancel Previous create roie | 


Figure 14-8. Creating a role 


249 


CHAPTER 14... USING SWARM MODE IN DOCKER CLOUD 


A new AWS IAM role called dockercloud-swarm-role is created, as shown in Figure 14-9. Click on the 
dockercloud-swarm-role role name. 


Deepak Vohra ~ Global ~ Support ~ 


Services v Resource Groups ~v Li 


4 Create New Role Role Actions * = o [?] 


Showing 1 results 





Dashboard 


Groups 
Iv Role Name $ Creation Time $ 
Users = 
| Roles N V)  dockercioud-swarm-role 2017-03-16 12 22 POT 
> 
Policies 


dentity providers 
Account settings 


Credential report 


Encryption keys 


Figure 14-9. New role 


Next, we will add an embedded (also called an inline) policy. The Permissions tab should be selected by 
default. Click on the v icon to expand the Inline Policies section, as shown in Figure 14-10. 


Ñ  Deopakvohra~ Global~ Support ~ 





Services ~v Resource Groups ~v * 


IAM > Roles > dockercloud-swarm-role 


‘ 

* Summary 
Dashboard Role ARN arn aws lam:67259835266535 role/doc kercloud-swarm-role 
Groups Instance Profile ARN(s) 
Users Path 

2017.03.16 12 22 POT 
| Roles Creation Time 2017-03-16 12:22 PO 

Policies Give this link to users who can hitps //signin aws.amazon.com/switchrole? —TA 
"Once 

switch roles in the console accountz672593526685&roleNamezdockercloud-swarm-role 


dentity providers 


Account settings ———— 
Permissions Trust Relationships Access Advisor Revoke Sessions 


Credential report 


Managed Policies 


Encryption keys 
There are no managed policies attached to this role 


Inline Policies Ye 
{ 


Figure 14-10. Expanding the inline policies 
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To start, no inline policies are listed. Click on the Click Here link to add an inline policy, as shown in 
Figure 14-11. 


Services ~ Resource Groups v *% lam Deopak Vohra~ Global~ Support ~ 





‘ - Summary 
Role ARN arn:aws.lam:. 672593526685. role'dockercloud-swarm-role 


Instance Profile ARN(s) 


Groups Path 

Users Creation Time 2017-03-16 12°22 POT 

Roles Give this link to users who can https //signin aws amazon com/switchrole? 

Pol switch roles in the console accounl=67259352668 5 &roleName-dockercloud-swarm-role 
Policies 


dentity providers 





Account settings Permissions Trust Relationships Access Advisor Revoke Sessions 
Credential report 

Managed Policies ^ 
Encryption keys There are no managed policies attached to this role 

Inline Policies ^ 


There are no inline polcies to show. To create one, click nere 
U 


Figure 14-11. Click on the Click Here link to add an inline policy 


In Set Permissions, select Custom Policy using the Select button, as shown in Figure 14-12. 


Sernices c Resource Groups ~ t O Deepak Vohra ~ Glob + Suppt v 





lanage Role Permissions Set Permissions 


Select a policy lamplate, generate a policy, or create a custom policy. A policy is a document that formally states one or more permissions. You can edit the policy 
on Hee following screen, or at a ater time using ihe user, group, or robe detai pages 


Policy Generator 
[*]Custom Policy 


use me poliey Gor tro Customize your own Ser oT permissions Select 


Figure 14-12. Selecting a custom policy 


A policy document lists some permissions and the policy document for an IAM role to use Docker 
for AWS may be obtained from https: //docs.docker.com/docker- for-aws/iam-permissions/. Click on 
Validate Policy to validate the policy, as shown in Figure 14-13. 


27070 


CHAPTER 14 ~ USING SWARM MODE IN DOCKER CLOUD 


Services + Hesource Groups ~ LI 





Manage Role Permissions Review Policy 


Deepak Vohra + Cabal + Support * 


Customize permissions by editing ihe folowing policy document. For mone information about the access policy language, see Cyerview of Policies in the Using 
IAM quide To test the effects of this pricy before appniying your changes, use the LAM Policy Simulator 


Policy Name 


dockerc loud-swarm-policy 


Policy Document 
"d 
"Version": "2012-10-17", 
"Statement": [ 
{ 


"gid": "5tet14819242399805", 


"Effect": "Allow" 
"Action": [ 
"cloudformation 


Ll 


"cloudfaormation 
"lt H — - T 
cloudformation 
"cloudformation 
"cloudfermation 
"cloudfermation 


"cloudfermation: 
"ecloudfermation: 
"eloudfermation: 


wj Use autotonmatting for policy editing 


Figure 14-13. Validating the policy 


ance lUpdateStack" 


Clee foaremad ran: 


r 
Cont invellpdatefol lback” 
CreateChange*Set", 
CreateStack" 


5 


P 


:CreatelploadBucket™, 
‘DeleteStack™ 


E] 

DescribeAccountlimits", 

DescribetchangeSet", 

DescribeStackEvents", 

Dear rihestackhesnurr e", T 


Cancel Validate Policy 
4 


Click on Apply Policy, as shown in Figure 14-14. 


Services ~ Resource Groups + LI 





Manage Role Permissions Review Policy 


Deepak Volira * Global ~ Support * 


Customize permissions ny edring the Thing policy Gecument For more information about the access policy language, see Cwerview of Policies in tne Lising 
LAM guide To test tne effects of this policy before apolying Your changes, use the LAM Policy Simulator 


This policy is valid 


Policy Name 


dockercloud-swarm-policy 


Policy Document 


"d 
"Version": "2012-10-17", 
Er "Statement": [ 


e { 
"Effect": "Allow", 

"Action": [ 
"cloudformation 
"cloudformation 
"cloudformation 
“cloudformation 


14 "cloudfermation 


af) Lise autoformatting for policy editing 


Figure 14-14. Applying the policy 
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"Sid": “Stmt14819241239905", 


:CancelUpdateStack", 
:Continuelpdateftollback", 
-‘Createthangeset” , 
ifreateStack”, 
"cloudformation: 
"eloudformation: 


CreatellploadBucket", 
leteStack", 


:DescribeRccountLimits", 
"eloudformation: 


DezcribechangeSet", 


cancel validate Policy Apply Policy In 


CHAPTER 14... USING SWARM MODE IN DOCKER CLOUD 


A new inline policy is added for the dockercloud-swarm-role role, as shown in Figure 14-15. 


Services + 


Despak Volira ~ Glabal + Support 7 





Resource Groups ~ * 


Creation Time 2017-03-16 12:22 PDT 





Give this link to users who can https-tsignin.aws amazon.corm/swilc hrole? Copy Link 
switch roles in the console account*6725932266 85 &rcleMame*dockerc aud. swarm-role 
Dashboard 
Groups 
T Permissions Trust Relationships AcCCeOSS Adviser Revoke Sessions 
sers 
Roles s 
Managed Policies A 
Policies 
Isentity providers There are no managed policies attached to this role 
Account settings Attach Policy 
Gredential report 
nline Policies ^ 


Encryption Keys 


This view Slows all ining policies that are emaedaoed in this role 


Create Role Policy 


Policy Hamme Actions 


ethene loud sani poney 
zi 


Figure 14-15. The new inline policy is added 


Copy the Role ARN String listed in Figure 14-16, 


provider from Docker Cloud. 


Services v 


Resource Groups v» LI 


Shew Policy | Edit Policy | Remove Policy | Simulate Policy 


as we need the ARN string to connect to the AWS Cloud 


Deepak Vohra ~ Global + Support ~ 





IAM » Roles » dockercloud-swarm-role 


- Summary 
Dasnboard b Role ARN arn.aws.iam..672593526685.role/dockercloud-swarm-role 
Groups Instance Profile ARN(s) 
users Path 
| Roles Creation Time 2017-03-16 12:22 PD? 
Policies Give this link to users who can https //signin.aws amazon.com/switchrole? y Lin 


Identity providers 
Account settings 


Credential report 


switch roles in the console accountz6725935265685&rcleNamezdockercloud-swarm-role 





Permissions Trust Relationships Access Advisor Revoke Sessions 


Managed Policies ^ 
Encryption keys 
There are no managed policies attached to this role 
Attach Policy 
nline Policies ^ 


Figure 14-16. 


T his view Shows all inline policies thal are embedoced in this role 


Create Role Policy 


Role ARN 
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Creating a Docker Swarm in Docker Cloud 


In this section, we create a Docker Swarm from the Docker Cloud service. Log in to the Docker Cloud service 
at https: //cloud.docker.com/. The Cloud registry page should be displayed at https: //cloud.docker.com/ 
app/dvohra/dashboard/onboarding/cloud-registry. A Swarm Mode option is available in the margin and 
it’s off by default, as shown in Figure 14-17. 


€ > C | @ Secure | https://cloud.docker.com/app/dvohra/dashboard/onboarding/cloud-registry 











sss | DOCKER 


a | CLOUD 


Welcome! 





Welcome to Docker Cloud! 


Let's get you familiarized with the central concepts of Docker Cloud. 


Figure 14-17. The Swarm Mode slider 


Click on the Swarm Mode slider; the Swarm mode should be enabled, as shown in Figure 14-18. 


€ > X | & Secure | https://cloud.docker.com/app/dvohra/dashboard/onboarding/cloud-registry tr | Bo 


\) Switching to Swarm mode... 





Figure 14-18. Switching to Swarm mode 


A Swarms toolbar option is added, as shown in Figure 14-19. 


> © | ü Secure | https//doud.docker.com/swarm/dvohra/dashboard/onboarding/cloud-registry 


E DOCKER me Ae 
o^ | CLOI = warm mode + Repositories Swarms Get Help ~ 9 dvohra + 


»* BETA 


Welcome! 





Welcome to Docker Cloud! 


Let's get you familiarized with the central concepts of Docker Cloud. 


Figure 14-19. Swarms toolbar option 
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Two options are available—Bring Your Own Swarm or Create a New Swarm. Click on Create to create a 
new Swarm, as shown in Figure 14-20. 


WT oa Docker Cloud 





D G | @ Secure | https /clowdidocker.comy swarm/dvohra/swarr/Tist/ 17page simez 10 ir 


+ Repositories Swarms Get Help ~ La dvohra = 
ATIA 








Bring Your Wn Swarm E 


A Swann is à cluster of Docxer Engines where you deploy services 


Click here 1o lear more 


@ 


Figure 14-20. Creating a new Swarm 


Next, we will configure the Swarm, including specifying a Swarm name, selecting a cloud provider, and 
selecting cloud provider options. Two Cloud service providers are supported: Amazon Web Services (AWS) 
and Microsoft Azure (not yet available). We use AWS in this chapter. We need to configure the cloud settings 
for AWS with the ARN string we copied earlier. Cloud settings may be configured with one of the two options. 
One option is to select Cloud Settings from the account, as shown in Figure 14-21. 


= Repositories Swarms Get Help ~ § dvohra + 
OETA 


O dvohra 


Swarm Name + Create Organization 





Account Settings 


Cloud Settings A 


Welcome Tour 
Service Provider 
Log Out 
mmi Microsoft 
^; amazon "d Azure 
web services™ 


Powered by Docker CE for AWS 


Docker Community Edition 


Docker Community Edition (CE) is ideal for developers and small teams looking to get started with 
Docker and experimenting with container-based apps 


Learn More 


Figure 14-21. Cloud settings 
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In the Cloud Settings page, click on the plug icon that says Connect Provider for the Amazon Web 
Services provider, as shown in Figure 14-22. 


= Repositories Swarms GetHelp ~ id dvohra ~ 
BETA 





Serle providers dohra 
Member since Oct 02, 2015 


Source providers ^ 





Naotificatians 


Default Privacy 

Service providers | Connect prover | 
Billing 

Amaren Web Services Add new credentials d A. Er r 
Pian 


Source providers 


Provider Account 
GitHub No account linked ; E] 
Bitbuckwet Ho account birked x E 


Figure 14-22. Connecting the provider 


The Add AWS Credentials dialog is displayed, as shown in Figure 14-23. 


Add AWS Credentials 


NOTE: Swarm mode requires a new policy to be attached to your AWS role. 


Click here for instructions. 





Figure 14-23. Adding AWS credentials 
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The other option to configure the Cloud settings is to click on the Amazon Web Service Service Provider 
icon, as shown in Figure 14-24, which also displays the Add AWS Credentials dialog. 





es Repositories Swarms GetHelp ~ Li dvohra + 
BETA 





Swarm Mame 


Service Provider 


ml Microsoft 


t 
amazon Azure 
amazon EE 


e not connected 
Powered by Docker CE for AWS 
Docker Community Edition 


Decker Community Edition (CE) is ideal for developers and small teams looking to get started with 
Decker and experimenting with contalner-based apps. 


Leam More 


Figure 14-24. Connecting to an Amazon web services provider 


Specify the ARN string copied earlier from the Add AWS Credentials dialog and click on Save, as shown 
in Figure 14-25. 


Add AWS Credentials 


NOTE: Swarm mode requires a new policy to be attached to your AWS role. 


Click here for instructions. 


arn:aws:iam::672593526685:role/dockercloud-swarm-role 





Figure 14-25. Saving the AWS credentials 
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With either option, the service provider Amazon Web Services should be connected, as indicated by the 
Connect Provider icon turning to Connected, as shown in Figure 14-26. 


a Repositories Swarms Get Help v 
BETA 





Genera 
Service providers dvohra 

Member since Oct 02, 2015 
Source providers "--— 


Notifications 


Default Privacy 

Service providers 
Billing 

Amazon Web Services arn:aws:iam::672593526685:role/dockercloud-swarm-role 4 9 Fr i 
Pian 


Figure 14-26. Amazon Web Services provider in connected mode 


The Amazon Web Services option should indicate connected, as shown in Figure 14-27. 


— Repositories Swarms GetHelp "* 


BETA 





Swarm Name 


Service Provider 


sg amazon ~ E i "à " 
Services" 

. © 

Powered by Docker CE for 
Docker Community Edition 


Docker Community Edition (CE) is ideal for developers and small teams looking to get started with 
Docker and experimenting with container-based apps. 


rm 


Figure 14-27. Amazon Web Services provider connected 
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Specify a Swarm name. That name should not include any spaces, capitalized letters, or special 
characters other than “,”, “-“ and" 5asshown in Figure 14-28. 


DOCKER 


nro 
ut 


CLOUD Swarm mo 


Create 





dvohra / DockerCloudSwarm 


No spaces, capitalized letters, and space 
characters other than ., , or - are allowed. 


Figure 14-28. Specifying a Swarm name 





Specify a valid Swarm name (docker-cloud-swarm), select the Amazon Web Services Service provider, 


which is already connected, and click on Create, as shown in Figure 14-29. 


4 Repositories Swarms Get Help v ? dvohra * 
BETA 





Swarm Name 
dvohra / docker-cloud-swarm 
Service Provider 
ml Microsoft 
amazon EUN Azure 
k 

e connected Coming soon 

Powered by Docker CE for AWS 
Docker Community Edition 


re eaman En a onan aans oola girias 
Docker and experimenting with container-based apps 


Figure 14-29. Creating a Docker Swarm using the AWS service provider 
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In the region, select a region (us-east-2), the number of Swarm managers (3), the number of Swarm 
workers (5), the Swarm manager instance type (t2.micro), the agent worker instance type (t2.micro), and 
the SSH key. Click on Create, as shown in Figure 14-30. 


Region 
us-east-2 v 
Region where your swarm is provisioned 
Swarm Size 
Number of Swarm managers? Number of Swarm worker nodes? 
3 v 5 ` 
Number of swarm manager nodes needs to be an odd number Number of swarm worker nodes (0 - 1000) 


Swarm Properties 


Swarm manager instance type? Agent worker instance type? 

t2.micro - t2.micro v 
EC2 HVM instance type (t2.micro, m3.medium, etc) EC2 HVM instance type (t2.micro, m3.medium, etc 
Which ssh key to use? 

docker v 


Name of an existing EC2 KeyPair to enable SSH access to the instance 
Create 


Figure 14-30. Configuring and creating a Swarm 


The Swarm should start to get deployed, as indicated by the DEPLOYING message shown in Figure 14-31. 


+ Repositories Swarms Get Help * 9 dvohra = 


BETA 





Bring your own swarm | Create | 


dvohra/docker-cloud-swarm 


O a few seconds npc 


DEPLOYING 


Figure 14-31. Deploying a Swarm 
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When the Swarm has been deployed, the message becomes Deployed, as shown in Figure 14-32. 


+ Repositories Swarms Get Help ~ 9 dvohra + 
BETA 





Bring your own swarm ES 


dvohra/docker-cloud-swarm 


ty DEPLOYED 


Figure 14-32. The Swarm is now deployed 


The AWS infrastructure for the Swarm is created and configured. A CloudFormation stack is created, as 
shown in Figure 14-33. 


(D CloudFormation ^ Stacks 





le aa | ACTIONS + Design template C 9 
Filter. Active = Showing 1 stack 
Stack Name Created Time Status Description 
x docker-cloud-swarm 2017-03-16 12:38 07 UTC-O7DU0 CREATE COMPLETE Docker far AWS 1.13 1 (ga-z) 


Figure 14-33. CloudFormation stack for the created Swarm 


A new proxy AWS IAM role for the Swarm is added, as shown in Figure 14-34. 





Services Resource Groups ~ 

sareh UAM i Create New Role Role Actions = ER o a 
Dashboard Showing 2 results 
Groups 

| | Rele Name £ Creation Time $ 

Users RO 
Roles = docker-cicud-swarm-ProxyRole-MISSEFAJQCOQTR 2017-03-15 12:38 PDT 
Policies [ | dockercloud-swarrn-rale 2017-03-16 12:22 PDT 


hdentty providers 
Account settings 


Credential report 


Figure 14-34. Proxy role and Docker Cloud Swarm AWS role 
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EC2 instances for the Swarm manager and worker nodes are started. Each EC2 instance is started with 


the proxy IAM role created automatically, as shown for a manager node in Figure 14-35. 


Launch Instance Connect Actions ~ 
| 4 





e 


? © 


| Q, Filter by tags and attributes or search by keyword Q kK 4 itesofs > > 
Name > Instance ID ^ Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status 
docker-cloud-swarm-worker i-Dübd2442d55f04ede 12 micro us-east-2b Q running © 2/2checks... None k 
E —deckercloud-swamrm-Manag.. — i-03H027044d22efb3 12. micro us-east-2a © running © 22checks ... None Er 
docker-cloud-swarm-Manag.. — i-D4fdab6cf8f51672d 12 micro us-east-2b @ running © B2checks ... None e 
docker-cloud-swarm-worker — i-05169d85603d643d6 — (2.micro us-east-2a Q running © 22checks... None “ys 
docker-cloud-swarm-worker i-067a12b88cOG044f0 — 12 micro Us-east-2c Q running @ 2i2checks ... None ^. 
docker-cloud-swanm-worker i-Dae14983fce64b264 2 micro us-east-2b Q unning © 2/2checks ... None b 
docker-cloud-swarm-worker i-Ubc 7fcf948af9 1805 I2 micro us-east-2c © running © 22checks ... None “Ss 
docker-cloud-swarm-Manag..  i-Ofdia015ad15203c2 12 micro us-east-2c @ running © 22checks .. None us 
b 
Platform - Network interfaces — eth) > 
IAM role — docker-cloud-swarm-ProxyRole- Source/dest. check True 


MISSEFAJOCHR 


Figure 14-35. IAM role for EC2 instances 


Each Docker Cloud account namespace must be associated with only one AWS IAM role. If multiple 
Docker Cloud accounts are to access the same AWS account, multiple roles must be created for each Docker 
Cloud account or Docker Cloud account namespace. Each AWS IAM role for Docker Cloud to access AWS is 
associated with an ARN string. The ARN string for a deployed Swarm may be edited with the Edit Endpoint 
link, as shown in Figure 14-36. 


E Repositories Swarms Get Help ~ $ dvohra ~ 
BETA 





Bring your own swarm 





Edit end 


dvohra/docker-cloud-swarm tp 
ies ag ^S B 


DEPLOYED 


Figure 14-36. Edit Endpoint link 
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If the Swarm endpoint is to be modified, specify a new ARN string (for a different IAM role associated with 
a different Docker Cloud namespace) in the Edit Endpoint dialog. Click on Save, as shown in Figure 14-37. 


Edit endpoint for dvohra/docker-cloud-swarm 


if left empty, Docker Cloud uses the auto-detected public IP of the manager nodes to connect to the Swarm. 
You can override this value by specifying an IP address or a hostname in the field below. 


docker-cloud-swarm-ELB-226413314.us-east-2.elb.amazonaws.com 


Cancel 





Figure 14-37. Editing the endpoint 


Next, we connect to the Docker Swarm. There are two ways to do so: 
e Connect directly from any Docker host 


e Obtain the public IP address of a Swarm manager from the EC2 dashboard and SSH 
login to the Swarm manager 


We discuss each of these options. 


Connecting to the Docker Swarm from a Docker Host 


Click on the Docker Swarm in the Docker Cloud dashboard. The Connect To dialog should be displayed with 
adocker run command, as shown in Figure 14-38. Copy the docker run command. 


Connect to dvohra/docker-cloud-swarm 


docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e 
DOCKER HOST dockercloud/client dvohra/docker-cloud-swarm 


Pro tip: Download Docker for Mac or Docker for Windows (Edge channel), login with your Docker ID, and 


connect to this swarm from your desktop. 
Close window | 


Figure 14-38. Listing and copying the docker run command to connect to the Swarm 
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Start an EC2 instance with CoreOS AMI, which has Docker pre-installed, as shown in Figure 14-39. 


1. Choose AMI 2 Choose Instance Type 3. Configure instance 4 Acad Storage 5 Add Tags 6. Configure Secunty Group !' Rewew | 


Step 1: Choose an Amazon Machine Image (AMI) Cancel and Exit | 


An AMI is a template that contains the software configuration (operating system, application server, and applications) required to launch your instance. You can select an AMI provided by — ^ 
AWS, our user community, or the AWS Marketplace: or you can Select one of your own AMIS 





Quick Start om = i 1 to 6 of 6 Products 
| C coreos x 
My AMIS 
t container linux Container Linux by CoreOS (Stable) 
AWS Marketplace 
fedes && (01112986 50 Previous versies | Sok by CoreOS 
Community AMIis $0.00 for software + AWS usage tees 
| Free ter etgbie | Lix/Ueix, Other 1298.5 0 | 64-04 Amazon Machine image (AMI) | Updated: N2/17 
* Categories CoreOS Contamer Linux automates software updates to ensure better sec unty and reliability of machines and 


containers running on large-scale clusters. Operating system updates 
All Categories 
More info 
Software Infrastructure (6) 


Figure 14-39. Creating an EC2 instance with CoreOS AMI 


Obtain the public IP address of the CoreOS instance from the EC2 console, as shown in Figure 14-40. 


Launch Instance Connect Actions ¥ 
g Heren instance geo 


Q, search: 0e525edBüb79fc52b0 © | Add filt Q K 4 1tetofi > > 
@ Name * Instance ID = Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ 
a Coros HÜe525edBb79fc52b0 —— t2. micro us-east-1c Q running à Initializing 
Instance: | i-0e525ed8b79fc52b0 (CoreOS) ^ Public DNS: ec2-34-207-220-127.compute-1.amazonaws.com BAC - 


Description Status Checks Monitoring Tags Usage Instructions 


Instance ID — i-De525ed8b753fc52b0 Public ONS (IPv4) — ec2-34-207-220-127 compute- 
1.amazonaws.com 
Instance state — running IPv4 Public IP 34.207.220.127 


Figure 14-40. Displaying EC2 instance detail 


SSH login to the CoreOS instance. 
ssh -i "coreos.pem" core@34.207.220.127 
Run the command copied earlier to connect to the Docker Swarm. 


docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e DOCKER HOST dockercloud/ 
client dvohra/docker-cloud-swarm 


290 


CHAPTER 14... USING SWARM MODE IN DOCKER CLOUD 


The dockercloud/client Docker image that’s used to connect to Docker Cloud is downloaded. 
A username and password prompt should be displayed. Specify the username and password for the Docker 
Cloud account in which the Swarm was created. 


Container Linux by CoreOS stable (1298.5.0) 


$ docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e DOCKER HOST 
dockercloud/client dvohra/docker-cloud-swarm 

Unable to find image 'dockercloud/client:latest' locally 

latest: Pulling from dockercloud/client 

b7f33ccOb4Be: Pull complete 

91b7430c5c68: Pull complete 

b686674c0e39: Pull complete 

19aaa30bba7a: Pull complete 

Digest: sha2S6: 11d3cc5e1a62c7324 |2a6e038 | ccffi9 ]53tc91d0b1c69c8D1d3b68629337558a6 
Status: Downloaded newer image for dockercloud/client: latest 

Use your Docker ID credentials to authenticate: 

Username: dvohra 

Password: 


A export command is output to connect to the Swarm. Copy the command. 


Use your Docker ID credentials to authenticate: 

Username: dvohra 

Password: 

-» You can now start using the swarm dvohra/docker-cloud-swarm by executing: 
export DOCKER HOST=tcp://127.0.0.1:32768 


Run the command. The Swarm is connected to the CoreOS Docker host. List the Swarm nodes using the 
docker node ls command. 


»export DOCKER HOST-tcp://127.0.0.1:32768 


»docker node ls 
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 


liuomlmb6n6xtq4apxayumsx3 ip-172-31-0-251.us-east-2.cornpute.internal. Ready Active 
bchea5x85m82jtzoq336trn8y ip-172-31-47-61.us-east-2.compute.internat. Ready Active 
e2b1785z5pqouakdceomdpsbi ip-172-31-42-130.us-east-2.compute.internal. Ready Active 
hzxb8choml.7gylaqtrjrh6phx ip-172-31-26-90.us-east-2.compute.internal. Ready Active 
pcnple9129w88ueonhdwUcoc ip-172-31-27-18.us-east-2.compute.internal. Ready Active 
rupjaojommfchjgcshffdobhf * ip-172-31-10-153.us-east-2.compute.internal Ready Active Leader 
uyl5xv7mhb6c8jamsofncplyh ip-172-31-25-137.us-east-2.compute.internal. Ready Active Reachable 
wi6zurda4nawf9mgku3enf6io ip-172-31-34-33.us-east-2.cornpute.internal Ready Active Reachable 
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Connecting to the Docker Swarm from a Swarm Manager 


The other option is to connect to a Swarm manager using its public IP address. First, we obtain the public IP 
address of a Swarm manager from the EC2 console, as shown in Figure 14-41. 


ese Connect Actions ¥ 
4 | Launch instance o 9 6 
Q o 1to80f8 
Name * Instance ID + Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status 
doc ker-cloud-swarm-worker D0bd2442d55fü4ede —— t2 micro us-east-2b @ unning Q 22 checks None % 
B) —dockercloud-swarm-Manag.. i03f1027044d22efb9 t2. micro us-east-2a © running © l2checks.. None ^w 
doc ker-cloud-swarm-Manag i-D4fdabbc (895 167 2d t2. micro us-east-2h Q running © 2/2 checks None ` 
doc ker-cloud-swarm-worker H05165d85603d643d6 — t2 micro us-east-2a Q running © 22 checks None b 
doc ker-cloud-swarm-worker i-D67a12b88c 0604470 t2. micro Us-east-Jc Q running © 22checks .. None wu 
doc ker-cloud-swarm-worker i-Dae14583fce54b764 2. micro us-east-Zb i running © 22 checks None b 
doc ker-c loud-swarm-worker i-Übc Ttc f3483f91805 t2.micro us-east-Jc Q running @ 22 checks None E 
doc ker-tloud-swarm-Manag iDMfidiadi5adiS203¢2 t2.micro us-east-2c Q running © 22 checks None E. 


Description Sais Checks Monitoring Tags 
Instance ID — i-036102 70440? 2efb3 Public ONS (IPv4) — ec2-52-14-145-223 us-east- 
2 compute amazonaws.com 
Instance slate — running b IPv4 Public IP 92.14. 146.223 
Instance type — i2. micro IPv6 IPs 


Elastic IPs Private DNS ip-172-31-10-153.us-east- 


Figure 14-41. Obtaining the public IP of a Swarm manager 


SSH login into the Swarm manager. 
ssh -i “docker.pem" docker@52.14.146.223 
The Swarm manager is logged in and the Swarm command prompt is displayed. 


[root@1locathost —]# ssh -i "docker.pem" docker@52.14.146.223 

The authenticity of host 52.14.146.223 (52.14.146.223)1 cant be established. 
RSA key fingerprint is e9:7f:d2:3c:de:6d:5d:94:06:e2:09:56:b7:2a:c6:9a. 

Are you sure you want to continue connecting (yes/no)? yes 

Warning: Permanently added '52.14.146.223 (RSA) to the list of known hosts. 
Welcome to Docker! 
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List the Swarm nodes using the docker node 1s command. 


Welcome to Docker! 

~ $ docker node l.s 

ID HOSTNAME STATUS 

AVAILABILITY MANAGER STATUS 

liuomlmb6n6xtq4apxayumsx3 ip-172-31-0-251.us-east-2.compute.internal Ready Active 

bchea5x85m82jtzoq336trn8y ip-172-31-47-61.us-east-2.cornpute.internal Ready Active 

e2bl785z5pqouakdceonidpsbi ip-172-31-42-130.us-east-2.compute.internal Ready Active 

hzxb8chomt7gyl.aqtrj rh6phx ip-172-31-26-90.us-east-2.compute.internai Ready Active 

pcnple9129w88ueenhdwflcoc ip-172-31-27-18.us-east-2.compute.internal Ready Active 

rupjaejommfchjgcshffdobhf * ip-172-31-10-153.us-east-2.compute.internal. Ready Active Leader 

uyl5xv7mhb6c8jain5sofncplyh ip-172-31-25-137.us-east-2.compute.internal. Ready Active Reachable 

wi6zurda4nawf9mgku3enf6ie ip-172-31-34-33.us-east-2.compute.internal Ready Active Reachabie 
Create a service using the docker service create command and list the service with docker service ls. 


docker service create \ 
--name hello-world \ 
--publish 8080:80 \ 
--replicas 1 \ 
tutum/hello-world 


The hello-world service is created. A Docker Cloud server proxy service is also listed. 
~ $ docker service create \ 
> --name hello-world \ 
> --publish 8080:80 \ 
> - - replicas 1 \ 
> tutum/hello-world 
hbiejbua8u5¢@skabun3dzkxk4 
~ $ docker service 1s 
TD NAME MODE REPLICAS IMAGE 
Ogzua3p56myx dockerdoud-server-proxy global 3/3 dockercioud/server-proxy: latest 


hbiejbua8u50 hello-world replicated 1/1 tutum/hello-world: latest 
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Bringing a Swarm into Docker Cloud 


Docker Cloud Swarm mode also has the provision to import an existing Swarm into Docker Cloud. The 
Swarm to be imported must have the following prerequisites: 


e Based on Docker Engine 1.13 or later nodes 
e Swarm manager incoming port 2376 unblocked 


In this section, we create a Swarm and import the Swarm into Docker Cloud. First, run the docker 
--version command to determine if the Docker host version is 1.13 or later. One of the EC2 instances 
provisioned by Docker for AWS may be used to create and import a Swarm, as the Docker version on the 
custom Linux distribution is > Docker 1.13; the node must be made to leave the Swarm before creating a new 
Swarm. Using the private IP address of the EC2 instance, initiate a new Swarm. 
docker swarm init --advertise-addr 172.31.23.196 

Copy the docker swarm join command output to join the worker nodes. 
~ $ docker --version 
Docker version 17.03.0-ce, build 60ccb22 
~ $ docker swarm init --advertise-addr 172.31.23.196 
Swarm initialized: current node (ylzc3h3slxO5ztbujtl3yf86p) is now a manager. 

To add a worker to this swarm, run the following command: 
docker swarm join \ 
--token SWMTKN-1-23snf1iuieafnydizzgf37ucwuz1.khg9atqsmysmvv6iw1.arw0-do29n83jptkkdwss5fjsd3rt V 


172.31.23.196:2377 


To add a manager to this swarm, run 'docker swarm join-token manager' and follow the 
instructions. 


Join a worker node on another EC2 instance with Docker 1.13 or later. 

docker swarm join \ 
--token SWMTKN-1-61gcsgkriildxz580ftdl3rq0s9p7h30ni2byktgvbd6y3dk7r-cpes7ofdsq8abhxtznh92tjrz \ 
10.0.0.176:2377 


The worker node joins the Swarm. 
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A Swarm with two nodes is created, as listed in the output to the docker node 1s command, which 
runs on the Swarm manager node. 


~$ docker node 1s 

HOSTNAME STATUS 

AVAILABILITY MANAGER STATUS 

trgb2t4ehs2gp3cjbrnqhs7a5 ip-172-31-6-64.us-east-2.compute.internal. Ready Active 
yl.ic3h3stxo5ztbujtl3yf86p ip-172-31-23-196.us-east-2.compute.internal Ready Active Leader 
~$ 


Next, import the Swarm into Docker Cloud. From the Swarm manager node, run the following 
command. 


docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock dockercloud/registration 
Specify the Docker ID at the username prompt and the password at the password prompt. 


~ S docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock dockercloud/ 
registration 

Unable to find image dockercloud/registration:latest? locally 

latest: Pulling from dockercloud/registration 

b7f33ccOb48e: Pull complete 

b52875cf8fd4: Pull complete 

23182c866468: Pull complete 

Digest: sha256: a3f39de96d2763b957e7bel22ce99b8lfbba03fbd6b2e54bd6071cafbelcabcl 
Status: Downloaded newer image for dockercloud/registration:latest 

Use your Docker ID credentials to authenticate: 

Username: dvohra 

Password: 


Specify a cluster name for the Swarm imported into Docker Cloud, or use the default. Specify cluster 
as dvohra/dockercloudswarm. The Swarm is registered with Docker Cloud. As for a Swarm created in the 
Docker Cloud Swarm mode, the Swarm may be accessed from any Docker host for which a command is 
output. 

Enter name for the new cluster [dvohra/wkhotlq8cw5u44x22qp6r4eau]: dvohra/dockercloudswarm 


You can now access this cluster using the following command in any Docker Engine 


docker run -rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e DOCKER HOST dockerctoud/ 
client dvohra/dockerctoudswarm 


To bring the Swarm into Docker Cloud, click on the Bring Your Own Swarm button in Swarm mode, as 
shown in Figure 14-42. 
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+ Repositories Swarms Get Help ~ ? dvohra * 


BETA 





Bring your own swarm 


k 


dvohra/docker-cloud-swarm 


REMOVED 


Figure 14-42. Bring your own Swarm 


The Swarm registered with Docker Cloud is added to the Docker Cloud Swarms, as shown in Figure 14-43. 


DOCKER mos 
CLOUD ine V 


Repositories Swarms Get Help ~ ? dvohra ~ 
BETA 


Swans 








Bring your own swarm | Create | 


dvohra/swarm-coreos 


UNAVAILABLE 


dvohra/dockercloudswarm 


© a few seconds ago ^ 
DEPLOYED 


Figure 14-43. Docker Cloud Swarms, including the imported Swarm 


Summary 


This chapter introduced the Docker Cloud Swarm mode, which is a managed service for linking the Docker 
Cloud managed service to a AWS service provider account and provisioning a Swarm from Docker Cloud. 
A Swarm created on the command line can be imported into Docker Cloud. In the next chapter we discuss 
Docker service stacks. 
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CHAPTER 15 


Using Service Stacks 





The Docker Swarm mode is Docker-native as of Docker 1.12 and is used to create distributed and scalable 
services for developing Docker applications. 


The Problem 


While single Docker image applications are also commonly used, a vast majority of Docker enterprise 
applications are comprised of multiple images that have dependencies between them. Docker Compose 
(standalone in v1 and v2) could be used to declare dependencies between microservices using the links 
and depends on options, but Compose (standalone) is archaic, other than the format for defining services, in 
the context of Swarm mode services. 


The Solution 


Docker Swarm mode has introduced service stacks to define a collection of services (Swarm mode services) that 
are automatically linked with each other to provide a logical grouping of services with dependencies between 
them. Stacks use stack files that are YAML files in a format very much like the docker- compose. yml format. 
There are a few differences such as the absence of links and depends on options that were used to define 
dependencies between microservices in Docker Compose (standalone). YAML (http://www. yaml.org/)isa 
data serialization format commonly used for configuration files. 

As of Docker v1.13, the docker stack subset of commands has been introduced to create a Docker 
stack. Using a stack file that defines multiple services, including services' configuration such as environment 
variables, labels, number of containers, and volumes, a single docker stack deploy command creates a 
service stack, as illustrated in Figure 15-1. The services are automatically linked to each other. 
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Stack 







docker stack deploy service2 


service3 


Figure 15-1. Service stack created with the docker stack deploy command 


Docker Compose versions 3.x and later are fully Docker Swarm mode compatible, which implies 
that a Docker Compose v3.x docker-compose. yml file could be used as a Stack file except for a few sub- 
options (including build, container name, external links, and links) that are not supported in a stack 
file. Docker Compose 3.x could still be used standalone to develop non-Swarm mode services, but those 
microservices are not usable or scalable with the Docker Swarm mode docker service group of commands. 
To use stacks to manage Swarm mode services, the following requirements must be applied. 


e Docker version must be 1.13 or later 
e Swarm mode must be enabled 
e Stack file YAML format must be based on Docker Compose v3.x file format 


To use service stacks, the Docker Compose version 3 YAML file format is used, but Docker Compose is 
not required to be installed. 

When using Docker Swarm mode, the Docker version requirement for Swarm mode is 1.12 or later. 
Before developing stacks to manage Swarm mode services, verify that the Docker version is at least 1.13. 
The Docker version used in this chapter is 17.0x. The docker stack group of commands listed in Table 15-1 
becomes available in Docker v1.13 and later. 


Table 15-1. The docker stack Commands 


Command Description 


deploy Deploys a service stack or updates an existing stack 
ls Lists the stacks 

ps Lists the Swarm mode tasks in a stack 

rm Removes a stack 

services Lists the Swarm mode services in a stack 
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Run the docker --version command to list the Docker version. To list the commands for stack usage, 
run the docker stack command. 


[root@localhost ~]# ssh -i "docker.pem" docker@34.205.43.53 
Welcome to Docker! 

~ $ docker --version 

Docker version 17.06.0-ce, build 02c1d87 

~ $ docker stack 


Usage: docker stack COMMAND 


Manage Docker stacks 


Options: 
--help Print usage 
Commands : 
deploy Deploy a new stack or update an existing stack 
ls List stacks 
ps List the tasks in the stack 
rm Remove one or more stacks 
services List the services in the stack 


To use stacks, the following procedure is used. 


1. Install Docker version 1.13 or later (not Docker version 1.12, which is used in 
several of the earlier chapters). 


2. Enable Swarm mode. 
3. Create a Stack file using Docker Compose (version 3.x) YAML format. 
4. Usethedocker stack group of commands to create and manage the stack. 


The chapter creates a service stack consisting of two services, one for a WordPress blog and another for 
a MySQL database to store the data in the WordPress blog. 


Setting the Environment 


We use Docker for AWS available at https: //docs.docker.com/docker-for-aws/ to launch a Docker 
Swarm mode cluster of nodes. Docker for AWS uses the AWS CloudFormation template to create a Docker 
Swarm mode cluster. Click on the Deploy Docker Community Edition (stable), shown in Figure 15-2, to 
launch a Create CloudFormation Stack wizard to create a Docker Swarm mode cluster. 





Deploy Docker Deploy Docker Deploy Docker 
Community Edition Community Edition Community Edition 
[CE] Mo [CE] for AWS [edge] [CE] for AWS [test] 
[sta 


Figure 15-2. Deploying the Docker Community Edition for AWS (stable) 
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Configure a Swarm using the Create Stack wizard as discussed in Chapter 3. You can specify the number 
of swarm managers to be 1, 3, or 5 and the number of Swarm worker nodes to be 1-1000. We used one 


Swarm manager node and two Swarm worker nodes, as shown in Figure 15-3. 


(D CloudFormation ~ Stacks > Create Stack 


Create stack 


select Template Specify Details 
| Specify Details 
- Spet ity a Stack name and parameter values You can use of change tne aetault parameter values, which are defined In the AWS CloudFormatian LT Pate. Learn more 
Options 
Review 
$5 Stack name | DockerSwarm 
Parameters 
Swarm Size 


Humber of Swann 1 + Humber of Swarm manager nodes (1,3. 5) 
managers? 





Humber of Swarm worker 2 Humber of worker nodes in mae Seam (0-1000 
nodes? 


Swarm Properties 
Figure 15-3. Configuring a CloudFormation stack 


The CloudFormation stack is created, as shown in Figure 15-4. 





(D CloudFormation ~ Stacks 
Actions * Design template c o 
Filter. Active > Showing 1 stack 
Stack Name Created Time Status Description 


2017-06-16 16:49:42 UTC-0700 CREATE_COMPLETE Docker for AWS 17.03. 1-ce (aws2) 


+ DockerSwarm 


Tags Stack Policy Change Sets mmprm 


Outputs Resources Events Template Parameters 


Overview 


Stack name; — DockerSwarm 
Stack ID: amr aws cloudformation:us-east-1:672593526585' stack DockerSwarm/7 76fa810-52ee-11e7 -9480-500c2170be62 


Status: CREATE COMPLETE 


Status reason: 


IAM Role: 


Description: Docker for AWS 17 03 1.ce (aws2) 


Figure 15-4. CloudFormation Stack for Docker on AWS 
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Three EC2 instances—one for Docker Swarm manager node and two for the Swarm worker nodes—are 
launched, as shown in Figure 15-5. The Linux distribution used by the CloudFormation stack is Moby Linux, 


as shown in Figure 15-5. 


1 


C) Instance State : Running Add filter 


Name Instance ID 


DockerSwarm-worker 
i —DockerSwarm-Manager 


DockerSwarm-worker 


i-01242202c7b019e28 
i-0bfc 93384db35835cd 


i-0d89f7a7e2121a09e 


a Instance Type ~ 


t2 micro us-east-1b 
t2.micro us-east-1b 
t2 micro us-east-1c 


inbound rules 


Scheduled events 
AMI ID 
3a81f12c) 


No scheduled events 


Moby Linux aws-v 17.03. 1-ce-aws2 (ami- 


Availability Zone ~ 


@ running 
Q running 
@ running 


Figure 15-5. The Moby Linux AMI used for Docker on AWS 


Instance State ~ 


o 9 © 
[2] 11030f3 

Status Checks ~ Alarm Status Public DNS (IPv4) 
© 2/2 checks None Ye ec 2-34-227-151-152.co 
© 22checks... None We  ec2-54-205-48-154.com. 
© 2checks... None We ec 2-34-227-191-199.co 
LI 

VPCID  vpc-4ccc6535 
Subnet ID — subnet-671cb6áb 


Before being able to use Docker on AWS, enable all inbound/outbound traffic between the EC2 
instances in the security groups used by the EC2 instances. This is shown for the security group for Swarm 
manager node instance inbound rules in Figure 15-6. 


Create Security Group Actions ¥ 
4 


Q, search: sg-1bbf7a6a Add filter 
Name Group ID 
$g-1bbf7a6a 
m sg-¢ Ibe 79b0 


Security Group: sg-c1bc79b0 


Description Inbound Outbound 


Edit 


Type ʻi 
b All traffic 
All traffic 


SSH 


+ Group Name 


DockerSwarm-NodeVpc SG-I 


DockerSwarm-ManagerVpc S 


Tags 


Protocol (i 


All 


VPC ID 
vpc-4ccc6535 
vpc-4ccc6535 


Figure 15-6. The security group inbound rules are enabled for all traffic 


o 9 0 
[2] 1to20f2 
Description 
Node Security Group 
Manager SecurityGroup 
mmprm 
Source (i 
0.0.0.0/0 
{0 
0.0.0.0/0 
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SSH login into the Swarm manager EC2 instance and obtain the public IP address from the AWS 
management console, as shown in Figure 15-7. 


Launch Instance Connect Actions Y 
fy Henen Instance | oto 


Q, Instance State : Running Add filter &> 1to30f3 
Name ~ Instance ID a Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 
Doc kerSwarm-worker i-01242202c 7b019e28 t2 micro us-east-1b " running © 2/2checks None u ec2-34-227-151-152 co 
@ —DockerSwarm-Manager übfc9384db35835cd —— t2 micro us-east-1b @ running © 2/2checks None Me ec 2-54-205-48-154 com 
Doc kerSwarm-worker i-0d8917a7e2121a09e t2 micro us-east-1c @ running @ 2/2checks None Me  0c2-34-227-191-199 co 


Instance: | i-0bfc9384db35835cd (DockerSwarm-Manager) Public DNS: ec2-54-205-48-154.compute-1.amazonaws.com mms m - 


Description Status Checks Monitoring Tags 


Instance ID — i-Obfc 9384db35835cd Public ONS (IPv4) = ec 2-54-205-48-154 compute- 
1.amazonaws.com 
nstance state — running Pv4 Public IP 54.205.48.154 
Instance type t2 micro IPv6 IPs 
Elastic IPs Private DNS — ip-172-31-19-138. ec2 intema 
Availability zone — us-east-1b Private IPs 172. 31.19 138 
Security groups — DockerSwarm-ManagerVpc SG- Secondary private IPs 
CONEBYSUT3E2, DockerSwarm- E 


Figure 15-7. Public IP address 


Using the key pair used to create the CloudFormation stack SSH login into the Swarm manager 
instance. 


ssh -i "docker.pem" docker@54.205.48.154 
The command prompt for the Swarm manager node is displayed. 


[root@localhost ~]# ssh -i "docker.pem" docker@54.205.48.154 
Welcome to Docker! 


List the nodes in the Swarm mode. 
docker node ls 
Three nodes, one manager and two workers, are listed. 
~ $ docker node ls 
ID HOSTNAME STATUS AVAILABILITY MANAGER 
STATUS 
bf4ifhh86sivqp030fzhk6c46 ip-172-31-21-175.ec2.internal Ready Active 


ozdhlojtnricny1y95xbnhwtq ip-172-31-37-108.ec2.internal Ready Active 
ud2js50r4livrqf3f4l30fv9r * ip-172-31-19-138.ec2.internal Ready Active Leader 
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Test the Swarm mode by creating and listing a Hello World service. 
docker service create --replicas 2 --name helloworld alpine ping docker.com 
docker service ls 
The docker service commands output indicates a Docker Swarm service, so it's created and listed. 


~ $ docker service create --replicas 2 --name helloworld alpine ping docker.com 
q05fef2a7cf98cv4Ar2ziyccnv 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE PORTS 
q05fef2a7cf9 helloworld replicated 2/2 alpine:latest 
~$ 


Configuring a Service Stack 


To create a service stack consisting of two services, one for a WordPress blog and another for MySQL database, 
create a stack file using the Docker Compose version 3 YAML format (https: //docs.docker.com/compose/ 
compose-file/). Create a docker-cloud.yml stack file (the filename is arbitrary) to specify two services 

(web and mysql) using Docker images wordpress and mysql respectively. Set the environment variables for 
the Docker images. The only environment variable required to be set is MYSQL ROOT PASSWORD for the mysql 
Docker image. The WORDPRESS DB PASSWORD environment variable for the wordpress Docker image defaults 
to the MYSQL_ROOT PASSWORD, but may also be set explicitly to the same value as the MYSQL_ROOT PASSWORD. 
Some of the other environment variables used by the wordpress Docker image are listed in Table 15-2. 


Table 15-2. Environment Variables for the Docker Image WordPress 


Environment Variable Description Default Value 

WORDPRESS DB HOST The linked database host, which is The IP and port of the linked mysql 
assumed to be MySQL database by Docker container 
default. 

WORDPRESS DB USER The database user. root 

WORDPRESS DB PASSWORD The database password. MYSOL ROOT PASSWORD 

WORDPRESS DB NAME The database name. The database is wordpress 
created if it does not already exist. 

WORDPRESS TABLE PREFIX Table prefix. Hi 


If we were to create a WordPress blog using the wordpress and mysql images with the docker run 
command, we would create Docker containers for each of the Docker images separately and link the 
containers using the -link option. If we were to use Docker Compose (standalone), we would need to add a 
links or depends on sub-option in the Docker Compose file. 
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Next, specify the Docker images and environment variables to the stack file for creating a service stack. 
To use the Docker Compose YAML file format for Swarm mode stacks, specify the version in the stack file as 
3 or a later version such as 3.1. The docker-cloud.yml file is listed: 


version: '3' 
services: 
web: 
image: wordpress 
links: 
- mysql 
environment: 
- WORDPRESS DB PASSWORD-"mysql" 
ports: 
- "8080:80" 
mysql: 
image: mysql:latest 
environment: 
- MYSOL ROOT PASSWORD-"mysql" 
- MYSOL DATABASE-"mysqldb" 


The ports mapping of 8080:80 maps the WordPress Docker container port 80 to the host port 8080. Any 
stack file options, such as links that are included in the preceding listing that are not supported by docker 
stack deploy, are ignored when creating a stack. Store the preceding listing as docker-cloud.yml in the 
Swarm manager EC2 instance. Listing the files in Swarm manager should list the docker-cloud.yml file. 
~$ ls -l 
total 4 
-IWXI-X--- 1 docker docker 265 Jun 17 00:07 docker-cloud.yml 


Having configured a stack file with two services, next we will create a service stack. 


Creating a Stack 


The docker stack deploy command is used to create and deploy a stack. It has the following syntax. 


docker stack deploy [OPTIONS] STACK 
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The supported options are discussed in Table 15-3. 


Table 15-3. Options for the docker stack deploy Command 
Option Description Default Value 


- -bundle-file Path to a Distributed Application 
Bundle file. An application bundle 
is created from a Docker Compose 
file just as a Docker image is created 
from a Dockerfile. An application 
bundle may be used to create 
stacks. Application bundles are an 
experimental feature at the time the 
chapter was developed and are not 
discussed in this chapter. 


--compose-file, -c Path to stack file. 

--with-registry-auth Whether to send registry False 
authentication information to Swarm 
agents. 


Using the stack file docker-cloud.yml, create a Docker stack called mysql with the docker stack 
deploy command. 


docker stack deploy --compose-file docker-cloud.yml mysql 


A Docker stack is created and the links option, which is not supported in Swarm mode, is ignored. Two 
Swarm services—mysql mysql and mysql web—are created in addition to a network mysql default. 


~ $ docker stack deploy --compose-file docker-cloud.yml mysql 
Ignoring unsupported options: links 


Creating network mysql default 


Creating service mysql mysql 
Creating service mysql web 


Listing Stacks 


List the stacks with the following command. 
docker stack ls 

The mysql stack is listed. The number of services in the stack also are listed. 
~ $ docker stack ls 


NAME SERVICES 
mysql 2 
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Listing Services 


List the services in the mysql stack using the docker stack services command, which has the following 
syntax. 


docker stack services [OPTIONS] STACK 


The supported options are listed in Table 15-4. 


Table 15-4. Options for the docker stack services Command 


Option Description Default Value 
--filter, -f Filters output based on filters (or 
conditions) provided 
--quiet, -q Whether to display only the IDs of the false 
services 


To list all services, run the following command. 
docker stack services mysql 
The two services—mysql mysql and mysql web—are listed. 


~ $ docker stack services mysql 


ID NAME MODE REPLICAS IMAGE 
ixvOoykhuoi4c mysql mysql replicated 1/1 mysql:latest 
vl7ph81hfxan mysql web replicated 1/1 wordpress: latest 


To filter the services, add the - - filter option. To filter multiple services, add multiple - -filter 
options, as shown in the following command. 


docker stack services --filter name=mysql web --filter name-mysql mysql mysql 


The filtered stack services are listed. As both services are specified using filter, both services are 
listed. 


~ $ docker stack services --filter name-mysql web --filter name-mysql mysql mysql 
1 


ID NAME MODE REPLICAS IMAGE 
ixvOoykhuoi4c mysql mysql replicated 1/1 mysql:latest 
vl7ph81hfxan mysql web replicated 1/1 wordpress: latest 


The services created by a stack are Swarm services and may also be listed using the following command. 


docker service ls 
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The same two services are listed. 


~ $ docker service ls 


ID NAME MODE REPLICAS IMAGE 
ixvOykhuo1i4c mysql mysql replicated 1/1 mysql:latest 
sl2jmsat3Oex helloworld replicated 2/2 alpine: latest 
vl7ph81hfxan mysql web replicated 1/1 wordpress: latest 


Listing Docker Containers 


The docker stack ps command is used to list the Docker containers in a stack and has the following syntax; 
output the command usage with the - -help option. 


~ $ docker stack ps --help 
Usage: docker stack ps [OPTIONS] STACK 
List the tasks in the stack 


Options: 
-f, --filter filter Filter output based on conditions provided 
- -help Print usage 
--no-resolve Do not map IDs to Names 
--no-trunc Do not truncate output 


To list all Docker containers in the mysql stack, run the following command. 
docker stack ps mysql 


By default, one replica is created for each service, so one Docker container for each service in the stack 
is listed. Both Docker containers are running on a Swarm worker node. 


~ $ docker stack ps mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

n9oqwaikd61g mysql web.1 wordpress:latest ip-172-31-37-108.ec2.internal 
Running Running 3 minutes ago 

infzi7kxg9g9 mysql mysql.1 mysql:latest ip-172-31-37-108.ec2.internal 
Running Running 3 minutes ago 


Using the -f option to filter the Docker containers to list only the mysql web.1 container. 


~ $ docker stack ps -f name-mysql web.1 mysql 

ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

n9oqwaikd61g | mysql web.1 wordpress:latest ip-172-31-37-108.ec2.internal 
Running Running 9 minutes ago 
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List all the running containers by setting the desired-state filter to running. 


~ $ docker stack ps -f desired-state=running mysql 


ID NAME IMAGE NODE 

DESIRED STATE CURRENT STATE ERROR PORTS 

n9oqwaikd61g mysql web.1 wordpress:latest ip-172-31-37-108.ec2.internal 
Running Running 10 minutes ago 

infzi7kxg9g9 mysql mysql.1 mysql:latest ip-172-31-37-108.ec2.internal 
Running Running 10 minutes ago 


Using the Service Stack 


Next, we use the stack to create a WordPress blog. The stack service called web may be accessed on port 8080 
on the Swarm manager host. Obtain the public DNS of the Swarm manager node EC2 instance, as shown in 


Figure 15-8. 


Launch Instance Connect Actions Y 
CL o 90 


Q, Instance State : Running Add fitter $2 1to3of3 
Name ~ Instance ID a Instance Type ~ Availability Zone ~ Instance State ~ Status Checks ~ Alarm Status Public DNS (IPv4) 
Doc kerSwarm-worker H01242202c7b019e28 t2 micro us-east-1b Q running @ 2/2checks None Ye — 0c2-34-227-151-152 co 
l6 — DockerSwarm-Manager MÜbfc9384db35835cd —— t2 micro us-east-1b Q running © 2/2checks None Mg ec 2-54-205-48-154.com 
Doc kerSwarm-worker H0d59f7a7e2121a09e t2 micro us-east-tc 3 running © 2/2 checks None % ec 2-34-227-191-199 co 
» 
Instance: | |-Obfe9384db35835ed (DockerSwarm-Manager) , Public DNS: ec2-54-205-48-154.compute-1.amazonaws.com SEE - 
Ww 
Description Status Checks Monitoring Tags 
Instance ID — :-0bfc9384db35835cd Public ONS (IPv4) ec2-54-205-18-154 compute- 
1 amazonaws com 


instance state — running IPv4 Public IP 54.205.48.154 


Figure 15-8. Public DNS of Swarm manager 
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Open the <public dns>:8080 URL in a browser. The <public dns>:8080/wp-admin/install.php URL is 
displayed to start the WordPress installation. Select Continue. Specify a subtitle, username, password, e-mail, 
and whether to discourage search engines from indexing the website. Then click on Install WordPress, as 
shown in Figure 15-9. 


[3 WordPress » Installation X 
€ Q @ Notsecure | ec2-54-205-48-154.compute-1.amazonaws.com:8080/wp- admin/install php?step - 1 
Welcome 


Welcome to the famous five-minute WordPress installation process! Just fill in the information below and you'll 
be on your way to using the most extendable and powerful personal publishing platform in the world 


Information needed 
Please provide the following information. Don't worry, you can always change these settings later 


Site Title Docker Swarm Stacks 


Username dvohra 


Jsernames can have only alphanumeric characters, spaces, underscores, hyphens, periods, and the © 
sym do 





Password seeeeeeeeeere a» Show 


Important: You will need this password to log in. Please store it in a secure location 


Your Email dvohra17G yahoo.com 


Double-check your email address before continuing 


Search Engine fi Discourage search engines from indexing this site 
Visibility it is up to search engines to honor this request 


Install WordPress fi 


Figure 15-9. Installing WordPress 
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WordPress is installed, as shown in Figure 15-10. Click on Log In. 


SE — 


/ [3 WordPress Installation X ' EA 





E @ | @ ec2-54-205-48-154.compute-Lamazonaws.com:8080/wp-admin/install. php?step=2 





Success! 


WordPress has been installed. Thank you, and enjoy! 


Username dvohra 
Password Your chosen password. 
Log In | 


Figure 15-10. WordPress is installed 


Specify a username and password and click on Log In, as shown in Figure 15-11. 


—M 


[| Docker Swarm Stacks« X 





€ C | (D Not secure | ec2-54-205-48-154.compute-1.amazonaws.com:8080/wp-login.php 





Username or Email Address 


dvohra 


Password 


eene | 
Remember Me Eod 





Figure 15-11. Logging in 
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The WordPress blog dashboard is displayed, as shown in Figure 15-12. 


Dashboard : Docker Sv x — e | (e mx 


€- CQ (D ec2-54-205-48-154.compute-1.amazonaws.com v + E 








m pe Dashboard 
Home 
Welcome to WordPress! — 
È Po - — 
Oh Media Get Started Next Steps More Actions 
B Pages E Write your first blog post BB. Manage widgets or mem 
i Comments - an About page © Turn comments on or 
hange eme el © U TL bout getting startec 
Appearance 
M^ Plug 
d, Users At a Glance " Quick Draft a 
# Tools wx iP a E 
Settings t m 
o ess 48 ) Twenty Seventeen theme 
earct e! waged 
" WordPress Events and News a 
Figure 15-12. The WordPress dashboard 
To add a new post, select Posts and click on Add New, as shown in Figure 15-13. 
Y Posts « Docker Swarm 5% X b eod 

€ CG © ec2-54-205-48-154.compute-1.amazonaws.com t EH 


cher Swarm St 





Figure 15-13. Adding a new post 
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In the Add New Post dialog, specify a title and add a blog entry. Click on Publish, as shown in Figure 15-14. 





['] Add Mew Post: Docker x 
c C (D ec2-54-205-48-154.compute- Lamazonaws.com: 30s 
WO Ai Doder swam Saca ++ *rw 


ZB Dashboard 


Add New Post 



































| mme Wordpress Blog with MySQL On Docker Swarm Mode Stack 


Permalink: Ripe ieni 205-38 - L5. compute Lamanon rs com;agg0 2017/02/27 /wondpress-DEog-w,. cCker-sarm-stgck; — tot 
Taua 


Oh Media 


Pages The wordpress blog is created using the Docker Swam Mode feature called 


"Stacks", which is available in Docker v1.13 and later 
Comments eee 


Appea: ance 
iY Plugins 
a Users 


Tools 


E Settings 





Figure 15-14. Publishing a new post 


Publish i 
Save Dran Perey 
T status: Draft Edit 

mo Visibility Public £d 


lr Publish immediately Ed) 


Move to Trash 


Format à, 


SAFE 


The new post is added. Click on View Post, as shown in Figure 15-15, to display the post. 


Edit Post» Docker Swarr H 
c X O 2c2-54-205-48-154.compute- Lamazonaws.com S080/wp-admin/pgost.php?posted&actionzedit 


Ld. View Post 





fü Docker Swan Stacks M 


Eb Dashboard 


Edit Post add ew 


Post published. view post 


All Posts 
Wordpress Blog with MySOL On Docker Swarm Mode Stack 
cups Permalink: ht5p://6c2-53-205-45- 154. compute - ]amazonaws.com:g080/ 201 7/05/17 /wond press blog-w,.-swarm-maode-stack/ | Eat 
| ác 
CEU LEGI s sua 
i Pages Paragraph " B I EE tt = = = e m2 = En 
L Comments 
The wordpress blog Is created using the Docker Swam Mode feature called 
J* Appearance “Stacks”, which is available in Dacker v1.13 and later 
Mr Plugins 
m Users 
# Took 
fal Settings 
4] = 
Waiting for ec2-548-205-48- 1 compote-La pepas Lact edited by dvohra on Pune 17, 2237 at 


Publish à 


Preview Changes 
} Status: Published Edi 
ae Visibiliy Public Edd 


in Published on: Jun 17. 2017 © 00:38 Edit 


Format a 


hien "à TUN 


e 3» Standard 
Aside 
image 
Vidgà 
Quote 


Lirik 


Gallery 





Figure 15-15. Viewing the new post 
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The blog post is displayed, as shown in Figure 15-16. 


[] Weedgress Beg with M, x hi | e | EF io 
i (P CD q€2-54- 205 48-154. compute-1.amazonaws.com:$080/2017/06/17 /wardpress- blog -with-mysal-on-docker-pwarm -mode-stack) Tr E f 


Ei Docker Swarm $tacks aF Customize Mi + New V Howdy, dvohra E 


DOCKER SWARM;STACKS 





JURE IT, £017 HY DVOHHA 


WordPress Blog with MySQL On Docker Search ... 
Swarm Mode Stack 





HECENT POSTE 

The wordpress blog is created using the Docker Swam Mode feature 

called “Stacks”, which is available in Docker v1.13 and later. WordPress Blog with MySOL On Docker Swarm 
Mode Stack 


Hello world! 


ES RECENT COMMENTS 


A WordPress Commenter on Hello world! 


Figure 15-16. Displaying a blog post 
Scroll down and add a comment, as shown in Figure 15-17. 


— — — O—— n — bead Lc | la 


O Wordpress Blog with M, X 
E (3 (D ac2-54- 205-48-154 compute- Lamazonaws com 8080/2701 7/06/17 wardpress-blog-with-mysqi -on«docker- warm-mode-stack/ t4 : 





ARCHIVES 
Leave a Reply 

june 2017 
Logged in as dvohra Log out? 
Comment CATEGORIES 


The Docker Swarm mode "Stacks" feature is available only in 
Swarm mode and in Docker v1.13 and later. 


Uncategorized 


META 


Site Admin 


Lag out 











Entries RSS 


Comments RSS 





WordPress.org 


PREVIOUS 


Hello world! 


Figure 15-17. Adding a comment 
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The comment is added, as shown in Figure 15-18. 


[5 Wordpress Blog with ki, x 


i C  ec2-54- 205-48-154 compute-1 amarzona.ws com BO80/ 2017/0641 7 wardpn 








VES gi Dodo Eom uk: c Nt IH TR 


JUHE L7, 2017 BY DVOHR A 


WordPress Blog with MySQL On Docker 


Swarm Mode Stack 


The wordpress blog is created using the Docker Swam Mode feature 


called "Stacks", which is available in Docker v1.13 and later. 


One Reply to "WordPress Blog with MySQL On Docker 


Swarm Mode Stack" 


dvohra 
JUNE 17. 2017 AT 12:41 AM EDIT 


The Docker Swarm mode “Stacks” feature is available only in Swarm mode and in 


Docker v1.13 and Eater. 


Figure 15-18. The comment has been added 


Removing a Stack 


Search... ey] 


RECENT POSTS 


WordPress Blog with MySQL On Docker Swanm 
Mode Stack 


Hello anor! 


RECENT COMMENTS 


dvohra on WordPress Blog with MySQL On 
Docker Swarm Mode Stack 


À WordPress Commenter on Hello world! 


ARCHIVES 


June 2017 


CATEGORIES 


The docker stack rm STACK command is used to remove a stack. Remove the mysql stack using the 


following command. 


docker stack rm mysql 


The mysql stack is removed and the docker stack service mysql command does not list the stack, as 


shown in the output from the command. 
~$ docker stack rm mysql 

Removing service mysql mysql 
Removing service mysql web 
Removing network mysql default 

~$ docker stack services mysql 


Nothing found in stack: mysql 
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Summary 


This chapter introduced stacks, a Docker-native feature added in Docker 1.13. A stack is a collection of 
related services and is created using a stack file, which is defined in YAML format similar to the Docker 
Compose v3.x YAML syntax. This chapter concludes this book about Docker management design patterns. 
As new features are added to Docker, other design patterns may be used for developing Docker-native 
applications. 
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